Parallel Processing Strategies of the Primate Visual System

Preface

Incoming sensory information is sent to the brain along modality-specific channels corresponding to the five senses. Each of these channels further parses the incoming signals into parallel streams to provide a compact, efficient input to the brain. Ultimately, these parallel input signals must be elaborated upon and integrated within the cortex to provide a unified and coherent percept. Recent studies in the primate visual cortex have greatly contributed to our understanding of how this goal is accomplished. Multiple strategies including retinal tiling, hierarchical and parallel processing and modularity, defined spatially and by cell type-specific connectivity, are all used by the visual system to recover the rich detail of our visual surroundings.

Introduction

Our richest life experiences and longest-lasting memories are formed through our interactions with the sensory world around us and have a profound impact on our personality and sense of self. Each of these sensations is transmitted to our brains through distinct biological machinery such as that found in the nose or under the skin. Yet, rather than perceiving the world as a disjointed collection of attributes, we most often experience a single unified percept. Parallel processing of sensory information is a commonly used strategy in the mammalian brain, not only between sensory modalities but across features of a single sense as well. Gasser and Erlanger¹ first demonstrated that the sensations of pain and temperature are transmitted through axons of different caliber than those transmitting touch. This work was shortly followed up by Bishop² who proposed that the three different classes of axons he found in the optic nerve process different sensory qualities related to vision. These early discoveries laid the foundation for our current understanding of the nervous system’s parallel processing strategies^3–8.

The need for parallel processing in the visual system is immediately appreciated when one considers the multitude of qualities that are present in the visual environment and the physical limitations of the way this information is initially encoded and signaled to the brain (Box 1). Colour, depth, shape and motion are just a few of the many dimensions by which we interpret our visual environment and generate appropriate behaviour⁹. Remarkably, this complexity in our visual surroundings is first encoded as a pattern of light on a two dimensional array of photoreceptors, with little direct resemblance to the original input or the ultimate percept. Within just a few hundred microns of retinal thickness, this initial signal encoded by our photoreceptors must be transformed into an adequate representation of the entire visual scene. This representation, however, must also be sufficiently condensed so that the axons carrying this information can pass through the optic nerve, which forms an anatomical bottleneck along the route from the eye to the brain. Probably owing to these constraints, incoming visual signals are processed by at least 80 anatomically and physiologically distinct neural cell populations and 20 separate circuits within the retina. These circuits comprise at least a dozen parallel pathways that project to the brain for further processing¹⁰. The visual cortex has the job of extracting the relevant information from this reduced signal and to further elaborate and integrate the information into a unified and coherent perceptual experience.

Box 1 From retinal input to cortical processing and perception

Visual input is initially encoded in the retina as a two dimensional distribution of light intensity, expressed as a function of position, wavelength and time in each of the two eyes. This retinal image is transferred to the visual cortex where primary sensory cues and, later, inferred attributes, are eventually computed (see figure). Parallel processing strategies are employed from the outset to overcome the constraints of the individual ganglion cell's limited bandwidth and the anatomical bottleneck of the optic nerve. Adapted from DeYoe and Van Essen (1988).

Many years of research have uncovered important details concerning the anatomy and functional organization of the primate visual system. This Review will focus on recent advances in our understanding of how primary visual cortex integrates parallel inputs, and constructs new, parallel outputs. These findings have been particularly helpful in elucidating the complex relationship between early parallel pathways of the retina and the processing streams in visual cortex. Instead of attempting to provide a comprehensive analysis at each level of the visual system, we will highlight key principles, such as retinal tiling, hierarchical and parallel processing and modularity (defined spatially and by cell type-specific connectivity), with the aim of providing a unified and coherent understanding of parallel processing strategies in the visual system. We hope that this will inform not only our understanding of visual perception, but of sensory processing in general.

Parallel Pathways from Retina to Cortex

The first steps in seeing begin in the retina, where a dense array of photoreceptors convert the incoming pattern of light into an electrochemical signal¹¹. The photoreceptor mosaic encodes the intensity of light as a function of position (two dimensions), wavelength and time (Box 1). Much information is lost from the outset, such as the exact spectral composition of the image. Nevertheless, computations carried out within our visual system are supplied with enough information to support highly precise hue discriminations and other perceptual abilities that inform our everyday behaviour. Remarkably, many of these computations are carried out within the retina, before visual signals even leave the eye. Specialized circuits extract basic sensory cues, such as spatial contrast and temporal frequency from the initial intensity distribution, and encode these properties across approximately 1.5 million ganglion cells which form the optic nerve that connects the eye to the brain.

Due to the anatomical bottleneck of the optic nerve, retinal output must be efficiently condensed. The strategy used by the mammalian visual system is to reduce the representation of the visual scene to a limited number of specialized, parallel output channels. Rather than send visual signals from the eye to the brain along a homogeneous population of ganglion cells, in the primate, at least 17 distinct ganglion cell types exist in the retina and at least 13 of these project in parallel to the lateral geniculate nucleus (LGN) of the thalamus and on to visual cortex¹⁰ (Figure 1). Each ganglion cell type is thought to tile the retina, providing a complete representation across the entire visual field of the primary sensory cues it conveys to the brain¹². These cues include different spatial and temporal frequencies and luminance and colour contrasts in the image. At any given point in the visual field, multiple ganglion cell types convey different aspects of the visual input simultaneously and in parallel to the brain.

Figure 1. Parallel processing in the retina.

(A) Over a dozen different ganglion cell types exist in the retina, each with their own distinct set of morphological features including soma size, dendritic-field size and density, and level of stratification within the inner plexiform layer (IPL). Top panel shows schematic cross-sectional representation of morphologically distinct retinal ganglion cell types exhibiting either monostratified or bistratified dendritic arborization in the IPL; the boundaries of the IPL are indicated schematically by horizontal lines. The vertical position of each schematic dendritic arbor indicates its characteristic stratification in the IPL. Lower panel shows top views of filled cells obtained using retrograde photostaining from rhodamine dextran injections in the LGN and superior colliculus; scale bar is 50µm. (B) Each ganglion cell type, such as parasol (left) or small bistratified (right) have a unique set of inputs from photoreceptor cells, horizontal cells, bipolar and amacrine cells. These retinal sub-circuits confer specialized physiological response properties to the ganglion cells¹⁴. (C) The receptive field mosaic of an actual population of ON-parasol ganglion cells (yellow; from Field and Chichilnisky (2007)) is overlayed on an example visual scene, not drawn to scale. Missing cells in the mosaic are likely to reflect experimental undersampling rather than gaps in the retinal representation. Each ganglion cell type tiles the retina so that at any given point in the visual field, multiple ganglion cell types (red, blue, and yellow ellipses) are present and signal complementary visual information simultaneously and in parallel to the brain. (A) Obtained from Field and Chichilnisky (2007), which was originally adapted from Dacey (2004).

Each ganglion cell type has its own distinct set of morphological features, such as soma size and dendritic field size and density¹³. Many of these features vary substantially as a function of retinal eccentricity, but at any given eccentricity allow for nearly unambiguous cell type classification. Each ganglion cell type also has a distinct pattern of dendritic stratification within the inner plexiform layer (IPL), allowing for highly specific patterns of synaptic connectivity with functionally and/or anatomically defined bipolar and amacrine cell types⁸ (Figures 1A, B). These bipolar and amacrine cell types, in turn, have their own unique connections with photoreceptors and horizontal cells in the outer plexiform layer (OPL), forming distinct anatomical circuits from photoreceptors to ganglion cells (Figure 1B). Finally, the soma of each ganglion cell type appears to be regularly spaced across the retina so that, collectively, their dendritic fields cover the retina uniformly and with constant overlap (Figure 1C). This highly ordered mosaic architecture holds up only among ganglion cells of the same type: subsets of a type do not cover the entire visual field and different types display no systematic spatial relationship¹². Together, these structural features suggest that each ganglion cell type is a fundamental unit of the retina and underlies a unique channel of visual information.

Three retinal ganglion cell types are particularly well characterized and have been linked to parallel pathways that remain anatomically segregated through the LGN and into the input layers and compartments of primary visual cortex (V1)^{5, 14} (Figure 2). Midget, parasol and bistratified ganglion cells comprise approximately 90% of all ganglion cells found in the primate retina and, together, they functionally complement each other to extend the range of vision in the wavelength and spatio-temporal frequency domains¹⁵. Midget ganglion cells are considered to be the origin of the parvocellular (P) pathway and comprise approximately 70% of the total population of cells projecting to the LGN¹⁴. These cells convey a red/green colour-opponent signal to the P layers of the LGN, which in turn project to layers 4Cβ and 6 of V1^{14, 16–19}. Cells in this pathway typically have small receptive fields, low contrast sensitivity, slow axonal conduction velocities and sensitivity to high spatial and low temporal frequencies. Parasol ganglion cells are considered to be the origin of the magnocellular (M) pathway and comprise approximately 10% of the total population of cells projecting to the LGN¹⁴. These cells convey a broadband, achromatic signal to the M layers of the LGN and on to layers 4Cα and 6 of V1^16–19. Cells in this pathway generally have large receptive fields, high contrast sensitivity, fast axonal conduction velocities and sensitivity to high temporal and low spatial frequencies. Finally, small and large bistratified ganglion cells make up at least part of the koniocellular (K) pathway and together comprise approximately 8% of the total population of cells projecting to the LGN¹⁴. These cells convey a Blue-ON/Yellow-OFF colour-opponent signal^{20, 21} to K layers 3 and 4 of the LGN, which in turn project to layer 1 and to the cytochrome oxidase (CO) blobs (described below) of layer 2/3 in V1^{5, 17, 20, 22, 23}. K cells in the LGN are defined as cells that express αCAM kinase and/or calbindin²² and are found not only in the intercalated K layers but also scattered throughout the M and P layers. This has made it difficult to characterize the full range of physiological response properties that are carried by the K pathway. For example, layer 4A of V1 receives a Blue-OFF/Yellow-ON colour-opponent input from the LGN¹⁷, but neither the locations of these LGN cells nor the type of retinal ganglion cell providing their input has been determined with certainty.

Figure 2. Parallel pathways from retina to cortex.

Midget, parasol and bistratified ganglion cells are well characterized and have been linked to parallel pathways that remain anatomically separate and distinct through the lateral geniculate nucleus (LGN) and into primary visual cortex (V1). Parasol ganglion cells project to magnocellular (M) layers of the LGN and on to layer 4Cα of V1 (gray). Midget ganglion cells project to parvocellular (P) layers of the LGN and on to layer 4Cβ of V1 (red). Small and large bistratified ganglion cells project to koniocellular (K) layers of the LGN and on to the cytochrome oxidase expressing patches (or blobs) of layer 2/3 (blue). Although these ganglion cell types are numerically dominant within the retina, many more types are known to exist and are likely to subserve important parallel pathways that are yet to be identified.

It is likely that many more parallel pathways exist between the retina and V1. The remaining retinal ganglion cells in this pathway are less numerous (12% of the total), but they consist of more types that are likely to provide diverse information about the visual scene. In fact, assuming that each ganglion cell type uniformly tiles the retina, most of the variation in numbers of ganglion cells for each cell type might be explained simply by dendritic field size: the larger the cell type’s dendritic field size, the fewer of those cells will be required to cover the entire retina¹². Whether these additional cell types maintain strict segregation in their connections through the LGN and into V1 remains unclear. Nevertheless, many studies have indicated functional and anatomical heterogeneity within the LGN that might be explained by additional, unidentified parallel pathways. Two prevalent types of visual response properties have been reported in the M layers of the LGN and three in the P layers^{24, 25}. Furthermore, there is some indication that the two dorsal, central and ventral K layers of the LGN each have their own characteristic response properties and projection targets within V1⁵. Finally, the rod-and cone-mediated responses of intrinsically photosensitive, melanopsin-expressing retinal ganglion cells have recently been characterized and suggest that this cell population may constitute a separate projection pathway to the visual cortex that conveys luminance and colour-opponent information²⁶.

The sensory cues that are carried by the parallel pathways bear little resemblance to our perceptual experience, and the functional role of each pathway is still poorly understood. Visual attributes such as motion, shape and colour must be computed from these sensory cues and integrated within cortex to create a unified and coherent percept. Lesion studies have shown that computing these attributes does not rely on one pathway alone. M lesions, for example, result in a large decrease in luminance contrast sensitivity for stimuli of high temporal and low spatial frequencies²⁷, and have little effect on colour contrast sensitivity^{27, 28} or speed and direction of motion discriminations. By contrast, P (and possibly K) lesions cause an almost complete loss of colour vision^28–30, reduce luminance contrast sensitivity for stimuli of higher spatial and lower temporal frequencies^{27, 28, 31}, and have little effect on shape discriminations^{30, 32}. These findings are consistent with the visual response properties of cells within each pathway, and highlight the fact that no single pathway appears to have a monopoly over any particular set of computations, such as those related to motion or shape perception. Ultimately, the key to uncovering the role of the early parallel pathways may require a better understanding of the relationship between these pathways and processing streams within the visual cortex.

Primary Visual Cortex and Parallel Processing Strategies

Once the condensed and parallel signals from the retina and LGN arrive in visual cortex, the original components of the visual scene must be extracted, elaborated upon and integrated into a unified percept. The visual cortex uses both hierarchical and modular processing in order to accomplish these goals³³. Although the basic tuning properties of cells do not differ substantially between the retina and LGN, within the first cortical synapses of V1 new and more complex information is extracted such as orientation, direction and colour selectivity⁶. These ‘inferred attributes’⁹ (Box 1) are organized into overlapping functional maps, with a columnar organization for orientation tuning, ocular dominance and visual space^{34, 35}. As visual information passes through V1 and on to extrastriate cortex, the response properties found within each subsequent area tend to increase in complexity and selectivity. New computations are carried out along the way, eventually leading to highly specialized areas concerned with object recognition and sensorimotor integration³⁶. Information flow within this hierarchical network is not unidirectional and dense feedback connections provide a substrate for recurrent processing³⁷. However, the functional details of this feedback system remain largely unknown and are beyond the scope of this review.

Most visual information from the LGN passes through V1 before being processed further in extrastriate visual cortex. Different strategies might be used within V1 to transfer parallel input signals into multiple output streams (Figure 3A). One possibility is that the parallel inputs stay separate within cortex and continue on past V1. A second possibility is that the parallel inputs converge indiscriminately within V1 and lack any organization or relationship with processing streams in extrastriate cortex. A third possibility, which current evidence favours, is that the parallel inputs converge within V1, but do so in an organized and specific way so that new parallel streams of information are systematically conveyed to the rest of visual cortex.

Figure 3. Cortical processing strategies.

(A) Multiple strategies might be used by a visual cortical area (rectangles) in order to transform parallel inputs (arrows to the left of rectangles) into multiple outputs (arrows to the right of rectangles). One possibility (top) is that segregation of inputs is maintained (arrows within top rectangle) and passed directly on to the outputs. A second possibility (middle) is that these inputs mix indiscriminately (arrows within middle rectangle) and bear no systematic relationship to the outputs. A third possibility (bottom) is that the parallel inputs converge in an organized and specific way (arrows within bottom rectangle) so as to form the basis for specialized outputs. (B) Early models of V1 proposed that M and P pathway inputs remained segregated within V1 as they passed through layers 4B and 2/3 respectively and on to extrastriate cortex. (C) Recent studies have provided evidence for extensive intermixing and convergence of M, P and K pathway inputs, suggesting that V1 outputs bear little or no systematic relationship to its parallel inputs. Cytochrome oxidase-expressing blobs are shown as blue circles.

Originally, it was thought that the early parallel pathways might maintain strict segregation within V1 and underlie separate processing streams in extrastriate cortex^{6, 38} (Figure 3B). The distinctive laminar organization of V1 and the alternating light and dark staining for the mitochondrial enzyme cytochrome oxidase (CO) in the superficial layers³⁹, which is correlated with the pattern of direct thalamic input to V1 due to the higher levels of metabolic activity in these regions, suggested a high degree of processing modularity and the possibility that parallel inputs to V1 remain well segregated. Furthermore, the functional parcellation of extrastriate cortex into separate processing streams (discussed further below), which suggested M- and P-like segregation of function well beyond V1, led Livingstone and Hubel to propose that the P pathway formed the basis for colour and form processing within the CO blobs (dark staining) and interblobs (light staining) in layer 2/3, and that the M pathway formed the basis for motion and depth-related processing in layer 4B⁶. This scheme was attractive in its simplicity and was supported by early studies suggesting a clean anatomical segregation of M and P inputs to layers 4B and 2/3 respectively⁴⁰, as well as a functional segregation of fast, achromatic responses in layer 4B and colour-opponent or orientation-selective responses in blobs and interblobs respectively^41–43. More recent studies, however, have called into question such a strict segregation⁴⁴.

Although the concept of modular function within V1 has stood the test of time, it is now clear that rather than maintain strict segregation, the early parallel pathways converge significantly within the first few synapses in V1 (Figure 3C). This is most evident in the connections from layer 4C to the CO blobs and interblobs of layer 2/3, which provide ample opportunity for mixing of early parallel pathways. The blobs and interblobs of layer 2/3 receive convergent input from M and P pathways, with the blobs receiving additional direct input from the K layers of the LGN^{22, 45–48}. Importantly, the K pathway was discovered after Livingstone and Hubel’s theory was first proposed, leaving its contributions to colour, form and motion/depth processing largely unaccounted for. Functionally, the intermixing of M, P and K pathways in V1 has been confirmed by showing that lesions of either M or P layers of the LGN can strongly affect the response properties of cells in both CO blobs and interblobs of V1^{49, 50}. It is therefore unsurprising that recent studies have often failed to find a clear relationship between the CO-stained compartments and form or colour processing^51–53, with several indicating that orientation and colour tuning are commonly found in the same V1 cells and across almost all layers^{52, 54, 55}. It is tempting to conclude from these data that early parallel pathways converge indiscriminately within V1 and lack any systematic influence on the computations that form outputs to the rest of the brain. Nevertheless, a closer inspection of V1 modularity at the level of specific cell types and their connections tells a different story.

Recent evidence suggests that functional modules in visual cortex are defined not only by laminar and spatial compartmentalization, but also by specialized connectivity (Figure 4). For example, in layer 4B of V1 (Figure 4A), cells from the M-recipient layer 4Cα form dense axonal branches, whereas cells from the P-recipient layer 4Cβ pass through layer 4B without branching. Likewise, pyramidal cells in the K-recipient layer 2/3 blobs have only sparse axonal branches in layer 4B^{45, 56}, suggesting that layer 4B is a segregated conduit of information from the M pathway, with little or no intermixing from the P or K pathways. However, layer 4B contains two types of morphologically distinct projection neurons: spiny stellates and pyramids. While the dendrites of spiny stellates are almost entirely confined to layers 4B and 4Cα, pyramids have an apical dendrite that extends through layer 2/3 and into layer 1. This places the apical dendrites of pyramids in a position to receive input from both P and K pathways in the superficial layers. Photostimulation studies in monkey V1 slices have confirmed that while spiny stellates receive input only from M-recipient layer 4Cα, pyramids receive substantial, though less dominant input from P-recipient 4Cβ as well⁵⁷ (Figure 4A). Analysis at the laminar level turned out to be misleading, as the majority cell type in layer 4B (pyramids) receives mixed M and P inputs and possibly K inputs as well.

Figure 4. Spatial and cell type-specific connectivity in V1.

(A) Layer 4B of V1 contains two excitatory cell types known as pyramidal (black, left) and stellate (black, right). Both of these cell types receive direct input from cells in M-dominated layer 4Cα (red), but only pyramids have apical dendrites that pass above layer 4B and into layer 2/3. The apical dendrites of these pyramids are in a position to receive inputs from P-dominated layer 4Cβ projections into layer 2/3 (blue). Mixed M and P inputs onto pyramids and M only inputs onto stellates has been confirmed in photostimulation studies on macaque monkey V1 slices⁵⁷. (B) Layer 3B contains pyramidal cells that project out of V1 (projecting pyramid) and those that remain within V1 (local pyramid). Projecting pyramids (left) receive input only from M-dominated layer 4Cα (gray arrow), whereas local pyramids receive mixed input from both M-dominated layer 4Cα and P-dominated layer 4Cβ (red arrow). Red X denotes lack of input from layer 4Cβ. (C) Outputs from V1 to V2 were originally thought to maintain strict segregation, with layer 4B of V1 projecting to the cytochrome oxidase (CO)-stained thick stripes of V2, and the CO blobs and interblobs of layer 2/3 projecting to the thin and pale stripes of V2, respectively (arrows). This spatial modularity of outputs has recently been called into question with evidence that layer 2/3 blobs and interblobs in V1 provide substantial input to the thin stripes in V2, and that all projection layers underneath interblobs, including layer 4B, project to both thick and/or pale stripes (dashed arrows with question marks). (D) Specialized and distinct populations of cells project from layer 4B of V1 to area MT (middle temporal, also known as visual area V5) or V2. Area MT receives input from a population of cells with large cell bodies and dense dendritic trees. The majority of these cells are stellates (80%), but a smaller number of pyramids (20%) also project to MT and are positioned preferentially underneath CO blobs where their apical dendrites can receive M inputs from layer 4Cα (red circles). V2 receives input from a population of cells with smaller cell bodies and sparse dendritic trees, the majority of which are pyramidal. Together, these anatomical specializations are consistent with layer 4B of V1 relaying a quick, M-dominated signal to MT and a more mixed M and P signal to V2⁷³.

Modularity based on cell types and connectivity has been supported by subsequent studies. Photostimulation data has shown that local neurons within layer 3 receive convergent input from both M-recipient layer 4Cα and P-recipient 4Cβ, but projection neurons within that layer receive direct input only from layer 4Cα (Figure 4B). These layer 3 projection neurons have densely tufted apical dendrites that distinguish them morphologically from neighboring local neurons⁵⁸. Many different cell types exist in layer 6 of V1 and each receives a unique combination of inputs from the other layers of V1⁵⁹. It appears that cell type-specific connectivity allows for systematic combinations of early parallel pathway inputs despite the misleading appearance of indiscriminate intermixing at the laminar or compartmental level (Figure 3, Figure 4). It remains unclear how these different connection patterns might underlie different visual response properties. Eventually, highly specialized functional properties found within V1, including the modularity of function proposed by Livingstone and Hubel, may be mapped onto the different cell types discussed above.

The same spatial and cell type-specific modules that allow for specialized intermixing of early parallel pathway input, also form the substrates for multiple output streams to extrastriate cortex (Figures 4C, D). Outputs from V1 to the second visual area (V2) have been particularly well studied. In V2, there is a repeating pattern of CO-stained compartments known as thick, pale and thin stripes, each with their own characteristic functional properties and afferent/efferent connection patterns^60–63. It was originally thought that layer 4B of V1 projected to the thick stripes of V2 and that the CO blobs and interblobs projected to the thin and pale stripes, respectively^{64, 65} (Figure 4C). However, recent studies using intrinsic optical imaging and/or improved anatomical tracers have shown that such a clean correspondence between compartments in V1 and V2 is unlikely. One of these studies has suggested that all projection layers underneath interblobs, including layer 4B, project to both thick and/or pale stripes in V2⁶⁶, while another study has proposed that layer 2/3 blobs and interblobs provide substantial input to thin stripes⁶⁷. Despite this apparent breakdown in the spatial segregation of outputs from V1 to V2, it is likely that at a submodular level, segregated outputs do indeed exist. There are increasing indications that each of the V2 stripe compartments can be further broken down into functionally specialized submodules, such as hue-selective maps in the thin stripes or disparity-selective clustering in the thick stripes^{68, 69}. Although the cell-type-specific connection patterns in V2 remain unknown, it is likely that these smaller functional compartments, or even specific cell types within them, receive segregated input patterns from V1. Inspection of V1 at the level of specific cell types and their connection patterns may also help us understand the degree of segregation in the outputs from V1 to V2, and the rest of visual cortex.

Output modularity based on cell type-specific connectivity has been demonstrated in the connections between layer 4B of V1 and extrastriate cortex (Figure 4D). Cells projecting to V2, the third visual area (V3) or MT (middle temporal, also known as visual area V5) are intermixed within layer 4B^70–72. It was unclear whether these different output streams were anatomically and/or functionally distinct at a sublaminar level. Early studies suggested that a majority of layer 4B cells projecting to MT are spiny stellate and, as such, anatomically distinct from other output cells within this layer of V1⁷¹ and that layer 4B cells projecting to MT are a separate population from those projecting to V2 ⁷². A recent study using a modified rabies virus that expresses green fluorescent protein (GFP) confirmed earlier indications that layer 4B cells projecting to MT are primarily spiny stellate and have larger cell bodies than layer 4B cells projecting to V2, but it also uncovered other important anatomical differences⁷³. Layer 4B cells projecting to MT are large cells with dense dendritic trees that lie close to the bottom of the layer. The majority of these cells are spiny stellate, but a substantial minority are pyramidal cells located preferentially underneath CO blobs. Layer 4B cells projecting to V2 are smaller with sparse dendritic trees situated throughout the layer and are mostly pyramidal cells located preferentially underneath CO interblobs. These anatomical specializations are consistent with these two cell populations conveying quick^{74, 75}, M-dominated signals to MT and more mixed M and P signals to V2. Therefore, it seems that early parallel pathways of the retina and LGN are recombined within V1 into both spatial and cell type-specific modules, forming multiple output channels that project to specific areas of extrastriate cortex.

Processing Strategies in Extrastriate Cortex and Beyond

The outputs from V1 and V2 to MT and visual area 4 (V4), represent the beginning of a more pronounced anatomical and functional segregation of signals known as the dorsal and ventral streams⁷⁶ (Figure 5). MT is specialized for processing motion and depth, whereas V4 is specialized for processing form and possibly colour^{36, 77–79}. These two areas reflect earlier segregation within the output modules of V1 and V2, as cells in layer 4B of V1 and the thick stripes of V2 project to MT, whereas the pale and thin stripes of V2 project to V4^{60, 62, 70, 71, 80}. Functional evidence provides additional support, as layer 4B of V1 and the thick stripes of V2 have a high proportion of direction- and disparity-selective neurons, whereas layer 2/3 of V1 and the pale and thin stripes of V2 consist mainly of neurons that are selective for orientation and colour. It would seem, therefore, that one of the goals of the systematic integration of parallel inputs that occurs within V1 and V2 is to construct new output channels that can support at least two new parallel streams of information flow.

Figure 5. Parallel processing streams of extrastriate cortex.

Dorsal (left of vertical, dotted line) and ventral (right of vertical, dotted line) streams constitute separate but interconnected pathways through occipital, parietal and temporal extrastriate visual cortex (rectangular boundaries). Vertical position indicates the hierarchical relationship between areas. Interconnections between streams can be found at essentially every level of the hierarchy. For simplicity, not all areas have been included and only those cross-stream connections mentioned in the text have been drawn. Dorsal and ventral processing streams subserve different behavioural goals, with the dorsal stream aimed at the visual control of skilled actions and the ventral stream aimed at object recognition. The very same sensory cues, such as motion, disparity and shape are processed along both processing streams, but within each stream distinct computations are performed on these same cues in order to support different behavioural goals. V1,Visual area 1; V2, visual area 2; V3, visual area 3; VP, ventral posterior; V3A, visual area 3A; MT, middle temporal; V4, visual area 4; V4t, visual area 4 transitional; VOT, ventral occipitotemporal; FST, fundus of superior temporal; PITd, posterior inferotemporal (dorsal); PITv, posterior inferotemporal (ventral); CITd, central inferotemporal (dorsal); CITv, central inferotemporal (ventral); AITd, anterior inferotemporal (dorsal); AITv, anterior inferotemporal (ventral); STPp, superior temporal polysensory (posterior); STPa, superior temporal polysensory (anterior); TF, temporal area F; TH, temporal area H; MSTd, medial superior temporal (dorsal); MSTl, medial superior temporal (lateral); PO, parieto-occipital; V6, visual area 6; PIP, posterior intraparietal; VIP, ventral intraparietal; LIP, lateral intraparietal; MIP, medial intraparietal; MDP, medial dorsal parietal; DP, dorsal prelunate; 7a, visual area 7a.

Anatomical studies have established that extrastriate cortex is indeed composed of at least two segregated but interacting parallel processing streams (Figure 5). The dorsal pathway consists of a large number of interconnected extrastriate cortical areas in the parietal cortex downstream of MT, including medial superior temporal (MST), fundus of the superior temporal (FST), superior temporal polysensory (STP), ventral intraparietal (VIP), lateral intraparietal (LIP) and 7a to name just a few^81–85. The apparent absence of substantial crosstalk between a dorsal-dorsal pathway through visual area 6 (V6) and the superior parietal lobule (SPL) and a ventral-dorsal pathway through MT and the inferior parietal lobule (IPL) indicates that the dorsal stream may actually consist of two relatively segregated subcircuits⁸⁶. The ventral pathway also consists of a large number of interconnected extrastriate cortical areas in the temporal cortex downstream of V4 including temporal areas F (TF) and H (TH) and the various subdivisions of infero-temporal (IT) cortex^87–89. It has been suggested that these parallel dorsal and ventral pathways maintain segregation all the way into motor-related frontal cortical areas such as the frontal eye field (FEF)⁹⁰. Likewise, within the dorsal stream, segregated inputs from the SPL to dorsal premotor area (PMd) and the IPL to ventral premotor area (PMv) have been shown to exist⁹¹. While dorsal and ventral streams clearly make up two relatively separate circuits, the anatomical segregation between the two streams is by no means absolute. There is clear evidence of crosstalk between streams, such as the reported connections between V4 and areas MT and LIP^{81, 82, 85}, as well as between anterior inferotemporal (AIT) cortex and areas FST, VIP and STP^{83, 84, 92}. New subdivisions of parietal and IT cortex continue to be established^93–95, and the specific connection patterns of these areas have, for the most part, reinforced the notion of segregated but interacting dorsal and ventral processing streams^{96, 97}.

Functional evidence indicates that the dorsal and ventral processing streams operate relatively independently as well. Neuronal processing along the dorsal stream is best characterized by direction of motion and binocular disparity selectivity in MT^{98, 99}, more complex motion analysis related to locomotion and pursuit/tracking in areas downstream from MT in the STS (MST, FST, and STP)^{100, 101}, and computations informing target selection for arm and eye movements, object manipulation and visuospatial attention in areas of the IPL (LIP, VIP, and V6). By contrast, neuronal processing along the ventral stream is best characterized by colour and contour selectivity in V4^{77, 78, 102}, more complex combinations of colours, patterns, and/or shapes in posterior inferotemporal (PIT) cortex^{103, 104}, and invariant representations of 2-D and 3-D shapes and objects in AIT. Lesion studies have further corroborated the observed physiological differences along the two processing streams, with dorsal stream lesions affecting smooth pursuit eye movements, speed discriminations, complex motion perception and the accurate encoding of visual space^105–109 and ventral stream lesions affecting orientation and complex shape discriminations, perceptual invariance and attention^110–112.

Although the existence of relatively separate and independent dorsal and ventral streams is now firmly established, our understanding of these two processing streams has undergone important revision in recent years. Ungerleider and Mishkin⁷ first proposed that dorsal and ventral streams mediated spatial and object related visual capacities, suggesting that each stream might process different visual attributes. In more recent years, however, a slightly different perspective has emphasized distinctions in the processing goals of each stream. In this view, the dorsal stream’s goal is to mediate navigation and the visual control of skilled actions directed at objects in the visual world, whereas the ventral stream’s goal is to transform visual inputs into representations that embody the enduring characteristics of objects and their spatial relationships¹¹³. Building on this conceptualization, Rizzolatti et al.⁸⁶ proposed that the two anatomically segregated subcircuits of the dorsal stream might mediate different behavioural goals as well: the dorsal-dorsal pathway concerned with the control of action ‘on-line’ (while the action is ongoing) and the ventral-dorsal pathway for space perception and ‘action understanding’ (the recognition of actions made by others). This framework implies that the visual attributes that are processed along parallel streams in extrastriate cortex might actually be similar, and that the important differences between the streams can be found in the manner by which these attributes are used to inform their respective goals. This would be consistent with the systematic intermixing of parallel inputs that occurs within V1, as the resulting outputs formed there, which subserve the dorsal and ventral streams, do not maintain exclusive rights over the attributes conveyed along any single input channel. Instead, these attributes are integrated in varying combinations and to differing degrees along each output channel, the details of which might be particularly relevant to the computations and behavioural goals of the dorsal and ventral streams.

Indeed, it has become clear that both the dorsal and ventral streams are likely to process the same set of visual attributes, but for different behavioural goals (Figure 5). Possibly, the best example of a visual feature that is processed along both dorsal and ventral streams is binocular disparity. Binocular disparity information is found in dorsal areas MT and MST^{114, 115}, as well as in ventral areas V4 and IT^116–118. In recent years, however, critical differences have been shown to exist in the way the two streams encode disparity and the types of behaviour such encoding schemes might inform. Sensitivity along the dorsal stream to coarse and absolute disparities, and to anti-correlated random dot stereograms (RDS), suggest that such processing may be used to spatially orient and guide actions^{119, 120}. By contrast, sensitivity along the ventral stream to fine and relative disparities, and reduced or absent responses to anti-correlated RDS, suggest that such processing might be closely linked to 3-D shape perception^{121, 122}. Disparity is not the only visual attribute that is processed in both dorsal and ventral streams, both shape and motion-related computations can be found throughout extrastriate cortex as well^123–127. Again, recent studies have shown important differences in the way these attributes are encoded and the behavior or perception they likely inform^128–130. Finally, recent studies have also shown that many extrastriate areas can become selective to stimulus properties that are not typically encoded by neurons in that area, such as the shape selectivity found in area MT following associative learning between directions of motion and static shapes¹³¹. Such findings challenge the idea that each visual cortical area provides a stable representation of the stimulus features it encodes and suggests that plasticity within cortex might allow behavioural context and the statistics of the visual environment to dynamically modify such representations.

As with V1, each extrastriate cortical area receives input from multiple areas upstream and sends outputs to multiple areas downstream. MT provides a good example of such an extrastriate cortical area, as it receives input from many sources in parallel (Figure 6). While the bulk of visual information to MT passes through layer 4C of V1, it has recently been discovered that MT receives at least two separate inputs streams that bypass this main route altogether. Sparsely scattered K cells in the LGN provide a direct, monosynaptic input to MT¹³² and a denser population of M and P cells provides a disynaptic input to MT via layer 6 Meynert cells in V1¹³³. The main ascending input through layer 4B of V1 both directly to MT and indirectly through V3 is dominated by 4Cα and the M pathway. Nevertheless, MT eventually receives P input through layer 4Cβ, likely via indirect input through V2¹³⁴. These data, along with numerous lesion studies²⁷, suggest that, MT and the dorsal stream rely heavily, but not exclusively, on visual information provided by the M pathway. Each of the input pathways to MT provides for varying degrees of M, P and K pathway convergence, probably providing visual information that is uniquely suited for specific computations and visual tasks.

Figure 6. Multiple input streams to area MT.

Multiple input streams exist from the lateral geniculate nucleus (LGN) to the middle temporal (MT) area. The major ascending input to MT passes through M layers of the LGN, and layers 4Cα and 4B of V1. V2 and V3 provide indirect inputs from layers 4Cα and 4B of V1, with V2 probably providing inputs from layer 4Cβ after a small number of additional synapses¹³⁴. Bypassing layer 4C altogether, a sparse monosynaptic projection from K cells in the LGN to MT and a disynaptic projection from M and P layers of the LGN through layer 6 Meynert cells in V1 to MT have both been identified¹³², ¹³³. Area MT is likely to use similar strategies to those found in V1 in order to process these parallel inputs and transform their signals into multiple output streams. The thickness of each arrow represents the strength of the connection. Tk, thick; Th, thin; P, pale.

What type of information might each of these parallel input pathways provide to MT? Few studies have been able to directly assess the function of an identified cell population or pathway within primate visual cortex. Movshon and Newsome¹³⁵ used antidromic stimulation to identify and functionally characterize neurons in V1 that project directly to MT. In a technically challenging set of experiments, the authors were able to identify and functionally characterize only 9 MT-projecting cells out of the 745 total cells recorded. Nevertheless, they reported a highly specialized and homogeneous population of direction-selective cells that are distinct from the general population in V1. Whether this direct input from V1 provides different information to MT than indirect inputs through V2 or V3 was investigated in awake, fixating monkeys in which the indirect pathway to MT through V2/V3 was inactivated by cooling the lunate sulcus¹³⁶. MT disparity selectivity was severely disrupted even as direction and speed selectivity remained largely intact. These results suggest that the direct pathway from V1 to MT provides speed and direction of motion information, whereas the indirect pathway provides disparity information. Although we know much less about other input pathways to MT and pathways that provide input to other extrastriate cortical areas, segregation of function along parallel inputs is likely to be similar in V4 and the rest of extrastriate cortex.

Ultimately, we must understand how each of these inputs is processed and integrated to form outputs to areas downstream. As the problem faced in extrastriate cortex is so similar to that faced in V1 (parallel inputs and multiple outputs), we might expect to find similar properties, such as modularity defined spatially and by cell-type specific connections. Indeed, within many extrastriate cortical areas there are clear examples of spatial modularity that are similar to those seen in V1 and V2. MT contains direction and disparity columns and clustered speed-tuning modules^137–139, and V3 contains modules for disparity and orientation^{140, 141}. There is also evidence that V4 may contain separate modules^142–144, possibly related to segregated processing of colour⁷⁷. Few studies have directly investigated the vertical organization of extrastriate cortical areas, but those that have find evidence for distinct computations performed within different cortical layers^{145, 146}. There is also evidence to suggest that these functional modules serve as a substrate for the formation of a new set of segregated outputs to be sent downstream¹⁴⁷. We have few details on these modules of extrastriate cortex, but there is strong evidence that they are ubiquitous and probably contain submodular organization and specialized connectivity similar to that seen in the compartments of V1 and V2.

Conclusions and Future Directions

Over the past fifty years, we have gained great insight into the ways by which the brain processes incoming visual inputs and recovers the information necessary to adequately inform our perception and everyday behaviour. It is clear that whether in the retina, V1, or beyond, parallel processing is a ubiquitous feature of the visual system. Several key principles have been elucidated regarding parallel processing strategies that will help us in understanding visual perception and sensory processing in general (Box 2). In the retina, over a dozen distinct ganglion cell types parse the incoming visual signals into functionally and anatomically specialized channels that project in parallel to the LGN and on to V1. Each of these ganglion cell types tiles the retina and provides a complete representation across the entire visual field of the attributes it conveys to the brain. Once in V1, these parallel input channels are recombined within modules that are defined spatially and by local connectivity, thus forming new parallel channels of information that can be sent on to the rest of the brain. The outputs formed in V1 and V2 lead to segregated, but interacting dorsal and ventral processing streams in extrastriate cortex. Both streams make use of a similar set of visual attributes, but perform different computations in order to mediate non-overlapping behavioural goals. Within each stream, however, each extrastriate cortical area probably uses the same strategies found in V1 to recombine and integrate multiple inputs and form new outputs to send downstream.

Box 2 Parallel processing in other sensory systems

Parallel processing is not only a ubiquitous property of the visual system, but it is found across the brain in all other sensory modalities as well. The multiple touch receptors underneath our skin and the different olfactory receptors in our nose are the beginnings of a similar process by which incoming sensory information is parsed into separate channels of information and eventually recombined to form a unified and coherent percept. Some of the properties discussed in this review may be unique to the visual system, but many are likely to be shared with other sensory systems. For instance, the mosaic tiling found along the body wall in the periphery of the somatosensory system is similarly structured to the tiling that occurs in the retina^153–155. There is also evidence for mosaic organization in the periphery of the olfactory epithelium in the olfactory system¹⁵⁶, but these mosaics do not show the same degree of spatial order as that found in the retina, probably due to the fact that, in some cases, they process essentially non-spatial modalities. The modularity that exists in visual cortex appears to be typical of sensory cortical areas in general^157–159, and in the few cases for which it has been studied, cell-type specific connectivity appears common as well^160–163. Finally, there is substantial support for the existence of hierarchically organized, parallel processing streams in each of the other sensory cortices^164–168.

Although much has been uncovered regarding parallel processing strategies in the visual system, many questions still remain. How many more parallel pathways exist between the retina and V1, and what are their anatomical and functional properties? How many different output pathways are formed in V1 and what is the functional contribution of each pathway to processing in the dorsal and ventral streams of extrastriate cortex? In order to answer some of these questions, methods that can directly relate detailed cell morphology and connection patterns to visual response properties and perception will be necessary. Some recently developed methods are already helping us on this endevour^{73, 148–152} (Box 3). Two photon microscopy, viral tracing and reversible inactivation or activation methods can all play a useful role in taking the next step in understanding parallel processing strategies in the visual system. Further research is necessary to fully understand how parallel processing strategies are used within and between each sensory modality so that we can enjoy our seamless perception of the world around us.

Box 3 Modern techniques to investigate parallel processing in the visual system

Numerous techniques have been developed to gain further insight into the mechanisms underlying parallel processing in the primate visual system^{73, 133, 134, 148–152}. Two photon calcium imaging. Using femtosecond pulsed laser light in combination with genetically-encoded or bulk-loaded fluorescent calcium indicators, it is possible to image the morphology and visual response properties of neurons deep within the intact brain. This technique offers the possibility of studying modularity of visual cortical areas at single-cell resolution, as well as correlating function with specific cell types and circuits¹⁴⁸.

Viral tracing. Genetically-modified viruses have become increasingly useful for tracing mono-and multisynaptic connections, and for introducing foreign genes into neurons of the primate brain. Rabies virus, in particular, has proven to be a powerful tool for uncovering multisynaptic connections of the primate visual system with the hope of elucidating the complex circuitry within and between different visual areas^{73, 133, 134, 150}.

Reversible Inactivation/Activation. Only recently, the possibility of activating and inactivating specific cell types and circuits has become a reality. By introducing and activating non-mammalian receptors in specific neuron populations of the primate visual system, these neurons are hyperpolarized or depolarized. For example, activation of the allatostatin receptor by the application of its ligand opens inwardly rectifying potassium channels and hyperpolarizes the neurons so that they are no longer visually responsive¹⁴⁹. Alternatively, the receptors Channelrhodopsin and Halorhodopsin can be activated by light and depolarize or hyperpolarize the neurons respectively^{151, 152}

Online Summary

• Multiple parallel processing strategies, involving over a dozen retinal ganglion cell types, can be found in the retina. Each ganglion cell type tiles the retina to provide a complete representation across the entire visual field of the visual attributes it conveys to the brain.

• Three retinal ganglion cell types have been particularly well characterized both anatomically and physiologically and project in parallel from the retina, through the lateral geniculate nucleus of the thalamus to the primary visual cortex.

• Primary visual cortex receives parallel inputs from the thalamus and uses modularity, defined spatially and by cell-type specific connectivity, to recombine these inputs into new parallel outputs.

• Beyond primary visual cortex, separate but interacting dorsal and ventral streams perform distinct computations on similar visual information to support distinct behavioural goals.

• Less is known about the parallel processing strategies used in extrastriate visual cortex. However, there are strong indications that these areas employ many of the same strategies found in primary visual cortex.

• Many of the parallel processing strategies found in the primate visual system are found in the other sensory processing systems of the mammalian brain.

Glossary

Percept

the perception that arises internally in the mind, based on an external stimulus, such as a visual stimulus

Photoreceptor

a specialized cell in the retina, which detects light and responds with a change in membrane potential and a change in neurotransmitter release

Tiling

relatively uniform and complete coverage of space

Hierarchical Processing

processing in serial order with more sophisticated properties emerging at higher levels by building up from simpler properties at lower levels

Parallel Processing

simultaneous processing of information through independent circuits

Modularity

Repeating modules are used to conduct similar operations. Typically, in the visual cortex, each module will perform an operation related to visual information from a portion of visual space. All of the modules together cover the space so that the operation would be conducted over the entire visual scene

Eccentricity

distance from the center. Typically used to describe the distance of a visual receptive field from the center of gaze and expressed as an angle, in degrees

Colour-opponent

a visual receptive field having the property of excitation in response to one colour and inhibition to another colour

Receptive Field

the location in visual space where a change in light can cause a change in neuronal activity

Broadband

typically used to describe visual receptive fields that are not colour opponent. Whether a cell is excited or inhibited by a stimulus at a particular part of its receptive field is relatively independent of the wavelength of light

Random Dot Stereogram

pair of random dot images that generate the sensation of depth when the eyes are positioned so that they focus at a location in front of or behind the images

Disynaptic

traversing two synaptic contacts. If a transsynaptic anatomical tracer were to spread from one neuron across synaptic contacts to a second neuron, the spread would be monosynaptic. If the tracer continued to spread from the second neuron, across more synaptic contacts, to a third neuron, then the spread would be disynaptic

Meynert Cell

a large projection neurons in the deep layers of the visual cortex

Antidromic Stimulation

used to determine whether a neuron recorded in one location projects to another distant location. Antidromic stimulation at the distant location generates action potentials that propagate back to, and are detected at, the recorded neuron

Fixation

maintenance of eye position at a particular location

Biographies

Jonathan Nassi is a postdoctoral fellow in the Department of Neurobiology at Harvard Medical School, working in the laboratories of Richard Born and Clay Reid. He received his B.Sc. in Cognitive Science from the University of California, Los Angeles. He received his Ph.D. in Neuroscience from the University of California, San Diego, where he investigated multi-synaptic connections in the visual cortex of non-human primates in the laboratory of Ed Callaway at the Salk Institute for Biological Studies. His current research is focused on relating identified anatomical circuits in the visual cortex to their response properties and behavioural function.

Ed Callaway is a Professor at the Salk Institute for Biological Studies in La Jolla, California. After graduate work studying neuromuscular development with David Van Essen at Caltech, he moved to the Rockefeller University, New York, where he did postdoctoral work with Larry Katz, focusing on development of the visual cortex. Present research in the Callaway lab is aimed at understanding cortical organization and function, particularly in the visual system. The lab also uses molecular biology and virology to develop novel tools for investigation of the structure and function of the nervous system.