What Is a Visual Stream?

Abstract

The dual stream model of the human and non-human primate visual systems remains Leslie Ungerleider's (1946–2020) most indelible contribution to visual neuroscience. In this model, a dorsal “where” stream specialized for visuospatial representation extends through occipitoparietal cortex, whereas a ventral “what” stream specialized for representing object qualities extends through occipitotemporal cortex. Over time, this model underwent a number of revisions and expansions. In one of her last scientific contributions, Leslie proposed a third visual stream specialized for representing dynamic signals related to social perception. This alteration invites the question: What is a visual stream, and how are different visual streams individuated? In this article, we first consider and reject a simple answer to this question based on a common idealizing visualization of the model, which conflicts with the complexities of the visual system that the model was intended to capture. Next, we propose a taxonomic answer that takes inspiration from the philosophy of science and Leslie's body of work, which distinguishes between neural mechanisms, pathways, and streams. In this taxonomy, visual streams are superordinate to pathways and mechanisms and provide individuation conditions for determining whether collections of cortical connections delineate different visual streams. Given this characterization, we suggest that the proposed third visual stream does not yet meet these conditions, although the tripartite model still suggests important revisions to how we think about the organization of the human and non-human primate visual systems.

INTRODUCTION

Leslie Ungerleider's (1946–2020) indelible influence on visual neuroscience is reflected in the dual cortical stream model of the non-human primate (NHP) and human visual systems. Earlier work had recognized the functional division between object and spatial perception, and it had been known, since at least the late 19th century, that two major white matter tracts extend rostrally from the occipital lobe through the parietal and temporal lobes. In a series of elegant primate lesion studies, Leslie showed that performance on tasks that recruited these two aspects of visual function, object and spatial perception, was selectively inhibited by removal of, respectively, portions of these occipitotemporal and occipitoparietal projections. These results, reviewed in her landmark article with Mort Mishkin (Ungerleider & Mishkin, 1982), provided the empirical basis for their influential model, in which the ventral “what” stream is specialized for representing visually discernible properties of objects and the dorsal “where” stream is specialized for representing visually discernable properties of the arrangement of objects in space.

Since its introduction, the dual stream model has undergone a number of revisions. The most well-known is the proposal of Goodale and Milner (1992) that the two streams (or “systems”) are better described by functional profiles that seemingly crosscut the object-spatial perception distinction. According to their model, the “vision-for-perception” ventral stream underlies conscious visual experience and represents visual information using an allocentric frame of reference, whereas the “vision-for-action” dorsal stream is unconscious and represents visual information using an egocentric frame of reference. Leslie also continued to update the model, which over time grew considerably both in its structural and functional complexity (Figure 1). The final major iteration of this process describes the dorsal stream as representing arbitrary and dynamic spatiotemporal relations to guide many forms of action like reaching, eye movements, and navigation, whereas the ventral stream represents the broad class of stable visual features of objects and scenes for cognitive processes like learning, memory, and decision-making (Kravitz, Saleem, Baker, Ungerleider, & Mishkin, 2013; Kravitz, Saleem, Baker, & Mishkin, 2011).

Figure 1. .

Versions of the dual visual stream model in the rhesus monkey brain. Shaded regions represent vision-related regions in the occipital, temporal, and parietal cortices. Color coding of regions as blue (dorsal) and red (ventral) follows that of Ungerleider (1995), which is an extension of the model depicted in (C). The same scheme was used to code the equivalent areas in (A) and (B). (A) The initial model of the dual visual steam (Mishkin, Ungerleider, & Macko, 1983). Arrows depict the ventral and dorsal streams, each beginning in area OC and splitting off in prestriate cortex (areas OB and OA) to reach into ventral inferior temporal cortex, or OTC (areas TEO and TE) and dorsally to the posterior parietal cortex, or OPC (PG). (B) An updated account of the dual visual stream model, incorporating forward and backward connections between the two streams (Ungerleider & Desimone, 1986). Solid arrowheads depict “forward” connections while line arrowheads depict “backward” projections. Solid arrow lines depict projections from peripheral and central representations alike, while dashed arrow lines depict projections only from peripheral field representations. “d” is meant to indicate that the projection from V1 to V3 only applies to area V3d. “m” indicates that V2 only projects to the medial portion of VIP. This model also includes projections to additional extrastriate areas such as areas VIP and MSTp in area PG and area FST (VIP: ventral intraparietal area, MSTp: medial superior temporal visual area, peripheral representation, FST: fundus of the superior temporal visual area). All definitions used in (A) apply to (B). (C) Another updated version of the model, including a proposed third visual branch in the superior temporal sulcus (STS) as well as many more projections (Distler et al., 1993). This model also emphasizes area TEO as responsible for a large degree of “cross-stream” projection between the ventral and dorsal streams. “d” is meant to indicate that the projections from V2 to V3 and FST to V3 only apply to area V3d. All definitions used in (A) and (B) apply (C).

In one of Leslie's last contributions to science, she proposed a different kind of revision of the model: that there is a third visual stream through motion selective regions of lateral occipitotemporal cortex (OTC) that is specialized for representing dynamic signals relevant to social perception (Pitcher & Ungerleider, 2021)—see also Deen, Schwiedrzik, Sliwa, and Freiwald (2023). This revision is notable given that the dual stream model—especially as characterized by Goodale and Milner—has been criticized in part based on how it functionally individuates different streams (Cloutman, 2013; de Haan & Cowey, 2011; Schenk & McIntosh, 2010). If there is now a third visual stream, one may reasonably wonder whether the dam has at last broken for the dual stream model, and the number of proposed streams may continue to proliferate to the point that the notion of “visual stream” is left theoretically shallow. Thus, this revision invites the question: What exactly is a visual stream, and how are different streams individuated?

We had always wanted to discuss this question with Leslie. In the most general terms, the notion of “visual stream” serves to map different aspects of visual function to different aspects of neural organization in the visual system. Part of why it is challenging to say more than just this is that it is an open secret in the field that many terms have been used, seemingly interchangeably, to describe functional bifurcations both within and between OTC and occipitoparietal cortex (OPC), including dual “mechanisms,” “pathways,” “streams,” and “systems.” For example, in Kravitz and colleague's update of the model, the dual “pathways” are described as themselves containing multitudes of different “pathways” (Kravitz et al., 2011, 2013). As we see it, an appropriate answer should therefore be taxonomic: Streams are not monolithic, but rather are to be situated relative to other levels of neural organization in the human and NHP visual systems. Any conditions for individuating between streams will then be a natural consequence of how they are positioned within this taxonomy.

In such situations, philosophy can be of utility to science, as a means of clarifying key theoretical notions (Laplane et al., 2019). Given our backgrounds and interest in philosophy of science, in this spirit, we here try to fix terms and differentiate between three interdefined notions—that of neural mechanisms, pathways, and streams—drawing on themes in the philosophy of neuroscience and taking inspiration from Leslie's body of work. As we characterize them, these notions stand in many-to-one ordinal relationships with each other, with streams the most superordinate of the three, and subordinate to the visual system as a whole. This provides a common taxonomy that can be used by scientists to make meaningful distinctions between terms that are often used in different ways by different researchers. Our proposed taxonomy also invites further conceptual refinement by neuroscientists and philosophers of neuroscience alike. As an illustration of its utility, we apply this taxonomy in critically discussing the proposed third visual stream.

The rest of the article is structured as follows. In the Why There is No Simple Answer section, we consider and reject a simple, nontaxonomic answer to our question based on popular “low-resolution” visualizations of the dual stream model, by highlighting three ways it misrepresents the underlying complexity of the visual system. In the Towards a Taxonomic Answer section, we propose a taxonomic answer that better reflects this complexity and also offers conditions for individuating visual streams relative to other levels of neural organization within the visual system. In the Three's a Crowd section, we critically assess some of the evidence that there is a third visual stream based on these individuation conditions. Although we believe the evidence does not clearly favor a tripartite stream model, it does nonetheless suggest other important shifts in how we conceptualize the organization of the visual system. The Conclusion section summarizes the article.

WHY THERE IS NO SIMPLE ANSWER

The visualizations of the dual visual streams introduced by Leslie and her collaborators (Figure 1A) seem to favor a relatively simple and straightforward answer to our question: Each stream consists of a dissociable series of hierarchically organized cortico–cortical connections starting in early visual cortex (EVC) with distinct functional profiles, which can be characterized in general terms as “what” versus “where” (or alternatively, “perception” vs. “action”). This answer is only viable if we assume that this characterization is a coarse-grained description of the dual streams that leaves out many details for the sake of idealization but would still broadly conform to a more fine-grained description of the underlying system. In contrast, we believe that the now textbook visualization of the streams was intended not as a coarse-grained but rather low-resolution depiction, as it was often paired with more detailed depictions that highlight features that conflict with the simple answer sketched above (Figure 1B and C). In this section, we highlight three such features of Leslie's dual stream model that are excluded from low-resolution depictions and so often go underappreciated or mischaracterized. First, we suggest that the visual system does not form parallel hierarchies but rather is better described as what we call a branching heterarchy. Second, we highlight how the streams are multifunctional with different branches for different visual functions. Third, we emphasize that the two streams are not informationally encapsulated from each other and instead engage in continuous cross talk. Because of these features, the simple answer is inadequate, and a more complex, taxonomic answer is needed.

The Visual Streams Form a Branching Heterarchy

The term “heterarchy” was introduced by McCulloch (1945) to describe intransitivity of values (e.g., a preference ordering of A > B, B > C, and C > A). The first instance we know of where the visual system is described explicitly as a “heterarchy” is the abstract of Barlow (1997, p. 1141), who calls it a: “multilevel heterarchy with connections acting in all directions.” Although suggestive, Barlow does not elaborate on what distinguishes a heterarchy from a hierarchy. Bechtel (2019) provides a characterization intended to capture the role of control mechanisms in complex biological systems. Following Bechtel's definition, we consider a heterarchy to be an ordering or leveling of elements of a complex biological system (e.g., brain regions) into stages in which there is large-scale violation of one or more of the following conditions necessary for a hierarchy: (i) transitive ordering of stages; (ii) each stage only projects to one element at a later stage; and/or (iii) there are fewer elements at later stages than earlier ones. Notably, this definition still entails there is some form of ranking of elements, but specifies the ways in which it is not hierarchical. Furthermore, this definition is distinct from the usage where “heterarchy” refers to multiple ways of ranking the elements of a complex system (Baker & Kravitz, 2024). By a branching heterarchy, we mean a system in which i–iii are all violated and specifically the number of elements at later stages grows from a smaller number of elements at the earliest stages. Although in principle there may be only a single element at the earliest stage, this is not inherent to the definition—a point we return to in the Three's a Crowd section.

Leslie often described the visual system as a hierarchy even when referring to the complexities depicted in Figure 1 (Ungerleider & Haxby, 1994), and so it is perhaps not a surprise that the visual system continues to be described in such terms—especially when it comes to the ventral stream (DiCarlo, Zoccolan, & Rust, 2012). However, our interpretation of this hierarchical characterization is that it was intended to emphasize that there is some kind of ordering in the stages of processing through the visual streams, which is also consistent with the idea of a processing heterarchy. For when we consider the pattern of connections in the model, the visual system structurally clearly forms a branching heterarchy (Figure 1C): There are a multitude of feedforward as well as reciprocal and nonreciprocal feedback connections between regions; connections often skip intervening regions; regions often project to a multitude of later regions; and the number of regions increase in number (or branches) from primary visual cortex onward. This is the first reason the simple answer is inadequate: It characterizes the streams as hierarchically organized when in fact they are heterarchical.

The Visual Streams Are Multifunctional

Leslie helped to popularize the idea that the visual streams can be associated with distinct overarching functional profiles like “spatial” versus “object” perception. Although useful as a shorthand, out of context, this masks the fact that each stream subsumed a variety of cortico–cortical (or cortico–subcortical) connections that served a variety of distinct visual functions (Figure 1B and C). For example, the revised model of Goodale and Milner (1992) emphasizes the role of the dorsal stream in reaching behavior rather than spatial vision, although Leslie had long recognized that the dorsal stream subserved both aspects of spatial perception and visuomotor control (e.g., Ungerleider & Desimone, 1986). That both streams are multifunctional is the focal point of the expansive models of the dorsal and ventral streams of Kravitz and colleagues (2011, 2013). These models identify a number of distinct cortical pathways associated with different visual functions that were associated with the streams throughout Leslie's work. We briefly summarize both of these models of Kravitz and colleagues here (see also Baker & Kravitz, 2024).

First, the dorsal stream model follows the familiar route from EVC to closely interconnected regions of the inferior parietal lobule (IPL) in OPC, which contains egocentric spatial maps and motion-selective middle temporal (MT)/medial superior temporal area (MST) and represents changes in visual information resulting from observer motion. From portions of IPL, there are then three main branches. The first two are a parieto-prefrontal branch that supports spatial working memory and self-directed eye-movements and a parietal-premotor branch that supports reaching and grasping of objects. To these, Kravitz and colleagues (2011) added the third parietal-medial temporal lobe (MTL) branch, which transforms incoming information from an egocentric to allocentric reference frames on to the hippocampus for the purpose of navigation. Second, the model of the ventral stream follows multiple parallel routes along the lateral and ventral surface of the temporal lobe that contain many reciprocal (and nonreciprocal) feedback connections that represent different kinds of object and scene qualities. From this thicket of connectivity, there are multiple cortico–subcortical routes to the neostriatum, ventral striatum, and amygdala in service of cognitive functions like reinforcement learning, value assignment, and visual learning. There are also multiple cortico–cortical branches to the MTL for representing landmarks and places, orbitofrontal cortex for reward evaluation and decision-making, and to ventrolateral pFC, which is involved in working memory and task switching. This is another reason the simple answer is inadequate: It characterizes the streams in terms of a single series of connections that serve general visual functions rather than as containing multiple branches that serve distinct visual and cognitive functions.

The Visual Streams Are Not Informationally Encapsulated

Common visualizations of the dual stream model tend to abstract away from functionally relevant direct connections between regions of the two streams (e.g., Figure 1). The pervasiveness of this sort of visualization has likely contributed to the misconception that the possibility of cross talk is a new or open question (Cloutman, 2013; Schenk & McIntosh, 2010). However, some form of cross talk between the streams has been a feature of many iterations of the dual stream model that Leslie introduced (Figure 1B and C). For example, beyond the connections depicted in Figure 1C, Leslie also identified a wide variety of connections from V4 to different subregions of IPL (Ungerleider, Galkin, Desimone, & Gattass, 2008).

Other lines of evidence, both anatomical and functional, provide further evidence of this continuous cross talk between the streams in humans. Anatomically, the streams principally follow the two canonical anterior–posterior white matter tracts, the superior longitudinal fasciculus and inferior longitudinal fasciculus. However, there are a multitude of tracts that directly connect virtually all stages of the dorsal and ventral streams, which have been revealed using human diffusion-weighted imaging (DWI) tractography and corroborated by ex vivo tissue dissection (for a review, see Bullock et al., 2019). Functionally, we have already seen one example of dorsal to ventral traffic, based on the projections from parietal cortex to MTL in the service of remembering scene information for navigation (Kravitz et al., 2011). Evidence also suggests that some aspects of object shape (e.g., structural descriptions and 3-D form) are represented dorsally, and this information is then passed to OTC (Ayzenberg & Behrmann, 2022; Premereur & Janssen, 2020). Regarding ventral to dorsal traffic, Milner (2017) reviewed several studies where quintessential “vision-for-action” tasks rely on information represented ventrally, including: grasping complex objects with more than one principal axis, grasping after a delay, grasping when multiple spatial relations are considered, grasping in depth under monocular viewing, and control of grip force based on weight. This is the final reason the simple answer falls short: It characterizes the streams as hoarding the information they represent, when in fact the information extracted from the visual signal along one stream is regularly shared with the other.

Before continuing, it is worth highlighting that the possibility of such cross talk has often been presented as a challenge to the dual stream model (Schenk & McIntosh, 2010). However, there is no inconsistency. To see this, it is helpful to import the idea of information encapsulation, which has traditionally been used to define functional modules in cognitive psychology (Fodor, 1983). For example, it has been claimed that part of what differentiates vision and cognition is that the former is a module that is informationally encapsulated from the latter (Pylyshyn, 1999; Fodor, 1983), and this in turn has motivated debate as to whether supposed top–down effects on vision undermine this claim (Firestone & Scholl, 2016). However, it is not obvious that encapsulation should be considered a necessary condition for individuating modules, which only requires that there are restrictions on what kinds of information is shared, not that no information is shared (Carruthers, 2006; Coltheart, 1999). Thus, the possibility of top–down modulation of early visual processing does not threaten our ability to differentiate between visual and cognitive brain areas (Ogilvie & Carruthers, 2016). The same considerations apply with respect to the dual visual streams (Henke, 2023). The fact that there are multiple routes of cross talk, and thus a lack of encapsulation, does not entail unrestricted communication between the streams or that the distinction between them should be dissolved.

TOWARD A TAXONOMIC ANSWER

At the most general level, the visual system may be thought of as all parts of the brain that are visually responsive, such as exhibiting some degree of visuospatial coding or retinotopy (or visually derived spatiotemporal coding; see the Three's a Crowd section). The dual stream has provided a model to compress and make comprehensible the structure of this system in terms of what kind of visual information goes where and why, starting in EVC. If, as we have just illustrated, the visual system does not form parallel hierarchies that serve general visual functions, and this is rather a feature of low-resolution depictions of its organization, then one might still reasonably ask whether it makes sense to talk about dual visual streams at all. Perhaps we should retire the model in favor of ones that rely on different organizing principles (de Haan & Cowey, 2011). Alternatively, the model may primarily have heuristic value—it is false but still useful (Goodale & Milner, 2018).

Contrary to these views, we believe the dual stream model does capture the detailed functional organization of the visual system once we provide a more explicit taxonomy of levels of organization in the visual system, both in terms of visual function and neural structure (Figure 1). Under our view, streams are the principal branches of the heterarchical organization of the visual system, which extend from EVC and are a level of organization subordinate to the visual system as a whole. Each stream is multifunctional, consisting of a variety of sub-branches, or pathways, that serve different visual functions that rely on shared information-processing resources. Pathways in turn are collections of co-located mechanisms that underlie the same visual functions. In this section, we take up each of these notions in turn, building up from mechanisms to pathways and then streams. From our characterization of streams, we also identify necessary conditions for individuating between streams, which we use to evaluate the evidence for whether there is a third visual stream in the next section (Figure 2).

Figure 2. .

How streams are situated in the proposed taxonomic hierarchy of neural organization. The width of the scheme indicates (in the abstract) the total extent of the visual system, which is subdivided (both structurally and functionally) into separate streams, pathways, and mechanisms. Red and blue indicate division of the system into two separate streams, with the opacity decreasing at each subordinate level. The number of divisions depicted at each subordinate level is the minimum number necessary for a neural system to subsume multiple streams, pathways, and mechanisms. This visualization does not capture how pathways or mechanisms might project from one stream to another, or how multiple pathways might serve different visual functions.

Neural Mechanism in the Visual System

One key aspect of explanation is to situate phenomena of interest in the causal structure of the world (Salmon, 1984). Often scientists do this by trying to discover mechanisms. However, this term is used in what may seem like different ways across branches of neuroscience (Ross & Bassett, 2024). Here, we follow the somewhat consensus minimal characterization that is common in philosophy of science, which is born out of attempts to characterize mechanistic explanation in many branches of science, including neuroscience (Craver & Darden, 2013). According to this definition a neural mechanism consists of: a set of neural components, the operations they perform, and how the components and their operations are organized. There is also the phenomenon, specified in terms of starting or input conditions and the resulting effects, that the mechanism is responsible for (Illari & Williamson, 2012). This notion is quite broad and so applies to phenomena at many different levels of neural organization, from how ion channels in the cell membrane of neurons explain membrane potentials to how communication across networks of brain regions explain performance on a visual task. Whatever the phenomenon of interest, mechanistic explanation tends to follow a strategy of decomposition and localization, in which a mechanism that is responsible for some phenomenon can itself be broken down into components that are, in turn, mechanisms (Bechtel & Richardson, 2010). In this way, the components of mechanisms at one level will decompose into other mechanisms at a lower level (e.g., a brain region decomposing into laminar mechanisms, which decompose into neurons that themselves contain mechanisms). In the present discussion, by “neural mechanism,” we will specifically refer to those where components consist of collections of brain regions that are responsible for different visual functions.

Under this characterization, OTC and OPC contain different neural mechanisms that underlie different visual functions. For example, the lesion and patient studies that formed the foundational evidence for the dual stream model follow the sort of interventionist logic that is considered paradigmatic for identifying the components of neural mechanisms that are responsible for different phenomenal effects (Craver, Glennan, & Povich, 2021). Neural mechanisms can also be related to the features of the two visual streams from the previous section: Multiple cortical mechanisms can also form heterarchies when combined (Bechtel, 2019); a mechanism can also pass information between other mechanisms found in the dorsal and ventral streams (Henke, 2023), and explaining different phenomena like visual object recognition and visually guided reaching can readily be characterized in terms of models that describe neural mechanisms in the ventral and dorsal visual streams, respectively (Ritchie & Piccinini, 2024; Henke, 2023).

Early instances of the dual models of the visual system identified the duality with contrasting “mechanisms” (Ingle, 1967). However, we do not think equating the dual streams with mechanisms is consistent with the minimal characterization of mechanisms summarized above (cf. Henke, 2023). Most obviously, OTC and OPC each underwrite a multitude of visual functions—that is, they underlie many visual phenomena—so they presumably contain many different neural mechanisms. However, a more subtle point is that although we may talk loosely about the mechanism for a phenomenon (e.g., a mechanism for object perception), mechanisms are always described relative to the contrastive effects (i.e., facets) of a phenomenon of interest. Because phenomena are multifaceted, there is strictly speaking no single mechanism that explains a phenomenon, but rather a collection of mechanisms that will explain their different facets, or rather, effects (Craver & Kaplan, 2020). For example, there is no single mechanism for visual working memory, but rather several that will ultimately depend on stimulus and task conditions (Christophel, Klink, Spitzer, Roelfsema, & Haynes, 2017). So, another level of organization is needed to describe how the collection of mechanisms that underlie the same visual function are related to each other.

Neural Pathways in the Visual System

Neural pathways have received less attention in the philosophy of neuroscience, but in our view, they are distinct from mechanisms. Whereas mechanisms are defined in terms of how their parts contribute to some specific facet of a phenomenon, pathways involve a web of causal connections in the absence of specific effects, which highlights the connectedness between brain regions rather than their detailed inner workings (Ross, 2018, 2021). Although they are distinct, we also believe that pathways and mechanisms are closely related. In the context of the dual stream model, our interest in patterns of connectivity is based on how they help explain the neural basis of different (visual) functions of interest. In light of this, we suggest that pathways consist of the organizational substrate that contains the different mechanisms that are responsible for the various facets of a phenomenon, and a mapping of the relationship between these mechanisms. Thus, in identifying pathways, our focus is not the particular effects of the phenomenon we are trying to explain, and so the inner workings of the mechanisms may indeed be less crucial. Rather, our focus is on how similar kinds of visual information are transmitted and to where, by different neural mechanisms, and so the notion of pathway is superordinate to that of mechanism.

Under this characterization, dorsal and ventral visual cortex can also be differentiated based on the neural pathways they contain. This is already reflected in the fact that each contains different white matter tracts that project from visual cortex to different downstream regions, and that following these patterns of connectivity is central to determining their visual function. Neural pathways can also be related to the features of the two visual streams from the previous section (Figure 1C): A pathway may have several branches in a heterarchy, may provide the basis for information sharing between streams, and be associated with several different visual functions, all of which are features of a more expansive models of the two visual streams (Kravitz et al., 2011, 2013). Although “pathway” and “stream” have been most consistently used interchangeably to describe the dual organization of the visual system, pathways under our usage are particular to patterns of connections that underlie specific visual functions; in other words, pathways, like mechanisms, are still defined relative to a visual phenomenon of interest. Importantly, this is distinct from the sense in which mechanisms are defined relative to phenomenal contrasts or effects, as any one visual phenomenon will be underwritten by multiple mechanisms and a variety of patterns of connections. Thus, OTC and OPC contain a multitude of pathways and once more a further level of organization is needed to capture the relationship between different pathways.

Neural Streams in the Visual System

Just as pathways provide a way of explaining the relationship between neural mechanisms that are responsible for (different facets of) the same phenomenon, the notion of stream serves to group together different pathways associated with different phenomena, but that exhibit structural and functional commonalities and/or interactions. Structurally, a stream picks out a collection of pathways that have common origin points for their inputs, project in similar directions through the brain, start out (partially) spatially collocated, and engage in cross talk with each other. Causal intervention in a stream may inhibit function of some or all of the pathways that it subsumes, depending on whether the disruption happens before or after branching (e.g., in OPC or projections from there to prefrontal or premotor regions). Functionally, the processes carried out by stages of pathways that are collocated may have similar neural tuning profiles (e.g., receptive field, visual feature selectivity, or frame of reference) and the phenomena that the pathways help explain are themselves relatable under a shared functional profile. So, we explain how visual functions are realized in the brain by describing neural mechanisms that are responsible for different facets of these phenomena, which are jointly localized to different visual pathways. In turn, streams group neural pathways that chart similar routes through the brain in the service of explaining distinct visual phenomena that have a common functional profile.

This notion of stream reflects the division made across multiple iterations of the dual stream model and captures the three main features that conflicted with the simple answer we considered. First, because the visual system forms a branching heterarchy, the two streams reflect the principal branches of this organization from EVC, which can each be characterized broadly in terms of the visual information they represent as well as their downstream connections (Kravitz et al., 2013). Second, these streams themselves consist of a multitude of pathways in the service of distinct visual functions. This fits the idea that streams contain a number of different visual pathways associated with different downstream projection zones and visual functions, such as the dorsal stream containing pathways for eye movements, reaching and grasping, and navigation (Kravitz et al., 2011). Each of these pathways contains the mechanisms that explain different facets of visual phenomena of interest such as object recognition or visually guided reaching (Ritchie & Piccinini, 2024; Henke, 2023). Third, the lack of encapsulation and pervasiveness of cross talk between the streams reflects the fact that other visual pathways serve to pass on visual information from one stream to another by transforming the representation of the information into the appropriate format (Milner, 2017; Kravitz et al., 2011).

This notion of a visual stream is restrictive in terms of what can “count” as a visual stream when applied to the visual system, both in structure and function. Here, we emphasize two potentially necessary conditions. The first is that a visual stream has distinct branches as origin points from EVC. As mentioned earlier, there are two canonical white matter tracts, inferior longitudinal fasciculus and superior longitudinal fasciculus, which project rostrally from EVC into regions of OPC and OTC, and from there split into branches that have multiple downstream termination points or follow noncanonical tracts that provide avenues for cross talk between these two principal tracks. The dual stream model is intended to reflect this initial partition, and any proposed stream must find its own separate course to run from EVC. The second is that a stream must contain a multitude of neural pathways that serve different visual functions that nonetheless rely on shared information-processing resources or a shared functional profile. Crucially, whatever this function is cannot be subsumed under another visual stream. As mentioned earlier, Kravitz and colleagues (2011, 2013) describe the dorsal stream in terms of the representation of arbitrary properties in a visuospatial reference frame, whereas the ventral stream represents the stable properties of aspects of visual objects and scenes. A useful compliment to this is suggested by Milner and Goodale (2008), who clarify that the ventral stream, by being in the service of “perception,” is in fact facilitating the planning of action whereas the dorsal stream functions in the service of executing action. This fits naturally with the proposal of Kravitz and colleagues in terms of what visual information is represented by each stream, and why. Any proposed further stream must similarly capture a multitude of visual functions that is not covered by these descriptions. In the next section, we consider whether these conditions are met by the proposed third visual stream.

THREE'S A CROWD

We now return to the proposal of Pitcher and Ungerleider (2021) that there are not two, but three, visual streams, with the third running laterally through OTC into the STS and specialized for representing the dynamic aspects of social perception. On the one hand, in principle, our characterization of streams is agnostic as to whether there are more than two visual streams. On the other hand, the various iterations of the dual stream model have attempted to capture different dualities in visual function and how they map onto dissociable (but intertwined) pathways and mechanisms in visual cortex. Thus, any proposed third visual stream must resist assimilation into this functional and structural duality. Instead, it must identify a branch of the visual heterarchy that, at the most superordinate level, is described as a stream (and not a mechanism or pathway). In this section, we critique the tripartite model based on the two individuation conditions that we proposed at the end of the previous section: (i) that there is a branch from EVC that represents a distinct form of visual information from that found in the two canonical visual streams; and (ii) that the stream subsumes a plurality of visual pathways that have a shared functional profile that is distinct from these two streams. We argue that the proposed third stream does not satisfy either condition, focusing on Pitcher and Ungerleider's discussion of motion-selective MT area and the role of STS in social perception. Instead, the proposed stream is better conceived as a separate pathway of the ventral stream (Rosenberg, Thompson, Doudlah, & Chang, 2023; Wurm & Caramazza, 2022; Kravitz et al., 2013). Nonetheless, their proposal suggests other important changes to how we think about the broadscale organization of the visual system.

MT as a Wellspring for Motion

Pitcher and Ungerleider point to the direct connection from V1 to motion-selective MT in monkeys as strong evidence for the beginning of the third visual stream, with white matter tractography supporting the presence of a similar route in humans. This V1-to-MT connection bypasses those that run more ventrally in OTC via V4. From MT, there are direct connections to more anterior motion-selective areas MST and fundus of the superior temporal visual area (FST), with FST then projecting to posterior STS (see, e.g., Figure 1C). Regions of STS are also tuned to motion and do not exhibit the same sort of contralateral visual field biases as ventral OTC. For example, such biases are observed in human MT and face selective regions of ventral OTC, but not in face-selective regions of the posterior STS (Sliwinska, Bearpark, Corkhill, McPhillips, & Pitcher, 2020). According to Pitcher and Ungerleider, this difference in contralateral field bias suggests that there is greater interhemispheric connectivity in STS compared with ventral OTC, which is consistent with a third visual stream representing the biological motion of objects across the full visual field. On the basis of these considerations, Pitcher and Ungerleider claim that the functional profile of the third stream is to represent dynamic properties of stimuli.

It is interesting to note that treating MT as the starting point of a third visual stream contrasts with two other characterizations that were also proposed by Leslie in earlier works. Many consider MT as a region of the dorsal stream (Kravitz et al., 2011; Born & Bradley, 2005), and Leslie also at times seemed to endorse this interpretation (e.g., Ungerleider, 1995; Ungerleider & Desimone, 1986). MT is closely connected to MST, which is one of the main regions of OPC that is considered constitutive of the dorsal stream, and it is closely integrated with portions of IPL that tend to be motion selective—although it also projects to posterior STS (Figure 1C). This makes sense, because many motion signals are important to dorsal stream function as they relate to tracking objects with eye movements or relate to observer motion (e.g., optic flow). Alternatively, MT has been treated as playing a role in both the dorsal and ventral streams. For example, this characterization is suggested by Boussaoud, Ungerleider, and Desimone (1990), and it is implicit in the dorsal and ventral stream models of Kravitz and colleagues (2011, 2013), because MT features prominently in both of them. That MT is also a region of the ventral stream is again suggested by Figure 1C (coloring in the figure being set aside), as it projects directly to V4 and both MT and FST project not just to posterior STS but also to TEO, which is one of the canonical regions of the primate ventral stream.

These alternative characterizations, as well as the evidence that supports them, are not considered by Pitcher and Ungerleider (2021) and make it difficult to consider MT the principal entry point into a third stream defined by motion selectivity. Because the role of MT in both the dorsal and ventral streams aligns with the proposed branching heterarchy model, one possibility is that this is because MT is not a region of extrastriate cortex common to both streams but rather a separate kind of early visual area (Rosa & Tweedale, 2005). There are several lines of evidence in support of this alternative model. Like V1 and other primary sensory cortical regions, MT matures early and independently in development (Warner, Kwan, & Bourne, 2012; Bourne & Rosa, 2006). Furthermore, MT receives direct connections from the koniocellular layers of the LGN, which are found between the magnocellular and parvocellular layers that project to V1 (Sincich, Park, Wohlgemuth, & Horton, 2004; Stepniewska, Qi, & Kaas, 1999). Recent results also suggest that at least in marmosets, lesioning V1 causes neurochemical and structural remodeling of existing connections from the magnocellular and parvocellular layers of the LGN to MT (Atapour, Worthy, & Rosa, 2022).

These findings suggest MT may be the wellspring for all spatiotemporal signals much as V1 is the cortical origin point for visuospatial ones, rather than serving as the beginning of a separate lateral stream—although the substantive projections from V1 to MT suggest V1 is still the dominant primary visual area. This possibility is consistent with our characterization of the visual system as a branching heterarchy, which does not presume a single component at the earliest stage of organization. It is also consistent with our hierarchy of organization in which streams are bundles of pathways with a common function profile that contain mechanisms that explain different visual functions. Because phenomena are multifaceted, the different mechanisms that underlie a phenomenon may have varied sources of inputs. We believe the possibility that V1 and MT provide separate inputs for different kinds of visual information to OPC and OTC is also consistent with the dual stream model. The possibility that visuospatial and spatiotemporal processing is somewhat dissociable in both the dorsal and ventral streams is readily consistent with them picking out different visual pathways of each stream, for example. Nor may MT be unique in this regard. For example, it has been proposed that V3A is the wellspring for 3-D vision for both streams, rather than being an exclusively dorsal region (Rosenberg et al., 2023). Going further, the functional duality that V1 and MT implies may even be elevated to the level of streams. Under such a model the dorsal and ventral streams are crosscut by the visuospatial and spatiotemporal streams—a “2 × 2” stream model, as it were. We are inclined to reject this proposal, because the functional profiles and patterns of connectivity related to processing spatiotemporal information was already inherent to the dual stream model from its earliest characterizations (Figure 1). However, we do not think such a proposal ultimately undermines the dual stream model, but rather provides a useful compliment and further illustrates the theoretical utility of the notion of visual stream.

STS and the Currents of Social Perception

Although MT does not appear to provide a separate origin for a third visual stream, this does not entirely rule out the possibility of a third stream unless it is taken as a strictly necessary condition for individuating streams. For example, other routes to the lateral surface and on to STS are still consistent with this model (Pitcher, 2021; Weiner & Gomez, 2021), and earlier characterizations of the dual stream model by Leslie often seemed to identify a third visual stream both in low- and high-resolution depictions even when the multiplicity of connections to and from MT were taken into consideration (e.g., Figure 1C; Distler, Boussaoud, Desimone, & Ungerleider, 1993). Thus, the case for a third stream can still be made based on the functional profile of STS and how it relates to those of the dorsal and ventral streams—although this would require giving up on the idea that streams are to be identified with principal branches from EVC.

Pitcher and Ungerleider (2021) point to several converging lines of evidence in favor of STS as a stream specialized for representing the dynamic aspects of social perception. First, face-selective regions in STS show greater responses in humans to moving than static faces, and this seems to be via a route that bypasses more ventral face-selective regions. There is also evidence that this difference in selectivity is greater in the right rather than the left hemisphere (Pitcher, Ianni, Holiday, & Ungerleider, 2023). A similar functional profile appears to exist in the face patch system of NHPs. Although these patches are all found in and around the STS in NHP, ventral patches are more selective for face form whereas dorsal patches are selective for natural motion. Second, human STS exhibits body-selective responses to videos of moving body parts, biological motion point-light stimuli, and agents performing actions. The most heavily studied body-selective region, the so-called extrastriate body area, is located in lateral OTC proximal to MT. Pitcher and Ungerleider conjecture that extrastriate body area is contained within the third visual stream. Again, in NHPs, portions of the STS also show greater selectivity to biological motion stimuli. Finally, on top of selectivity for the motion of faces and bodies as such, Pitcher and Ungerleider highlight that STS responds to visual and auditory cues that have social significance such as facial expression and eye gaze, and voices and audiovisual integration. Adjacent and superior to posterior STS is also the TPJ, which is strongly associated with theory of mind—although we note TPJ also has a complex structure with multiple subregions with different functional profiles (Schurz, Radua, Aichhorn, Richlan, & Perner, 2014).

It is well-established that the STS plays a critical role in representing dynamic visual signals relevant to social (and nonsocial) encounters. However, the question at hand is whether we should consider the STS to constitute a separate visual stream, given the second individuation condition we identified in the previous section: namely, a series of distinct pathways that serve different visual functions but that also have a shared profile in terms of the type of visual information that is represented. We argue against this proposal for two reasons. First, the evidence surveyed by Pitcher and Ungerleider is also consistent with the thesis that STS contains lateral pathways for representing the dynamic features of faces and bodies of other agents, and these are separate from those along the ventral surface of OTC that represent static features. In which case, STS contains those pathways that contain mechanisms that explain some facets of the broader phenomenon of representing faces and bodies, in line with existing models (Duchaine & Yovel, 2015; Peelen & Downing, 2007). Second, as we have seen, the broad functional profile of the ventral stream is to represent any stable properties of objects and scenes relevant to different cognitive processes (Kravitz et al., 2013). According to this model of Kravitz and colleagues, this functional profile subsumes stable dynamic features as well, which is why STS is included in their more inclusive models of the ventral stream, to include both visuospatial and spatiotemporal processing. Thus, the STS can equally well be described as a pathway of the ventral stream rather than an additional stream itself (Wurm & Caramazza, 2022). For a more in-depth study of the functional profile of STS and its role in social cognition, which is also inspired by Leslie's work, see Puce (2024).

Even if we find the evidence that STS contains a third visual stream lacking, we would like to highlight what we interpret to be an important, but more subtle aspect of Pitcher and Ungerleider's model. Their proposal primarily relies on evidence of face and body selectivity in the human and primate brains, which is usually interpreted as reflecting distinct networks specialized for representing these stimuli (Duchaine & Yovel, 2015; Peelen & Downing, 2007). However, their proposal deviates from this more common interpretation in two important respects. First, they emphasize these responses as they apply to organisms and agents. This is more consistent with the view that the specialization is not for faces and bodies as such, but for persons and agentive organisms as a whole (Taubert, Ritchie, Ungerleider, & Baker, 2022; Hu, Baragchizadeh, & O'Toole, 2020). Second, the framing they offer is not that there is a visual stream for representing dynamic signals related to persons, but rather for social perception. Faces and bodies are often associated with distinct forms of “category-selectivity,” which centers the importance of representing these stimuli and the process of object and scene categorization (Grill-Spector & Weiner, 2014). However, it has been suggested that the field shift its theoretical perspective away from focus on categorization and instead consider the neural responses to such stimuli based on the behavioral goals they serve (Bracci & Op de Beeck, 2023; Peelen & Downing, 2017; Malcolm, Groen, & Baker, 2016). This also applies to social interaction (Deen et al., 2023). As Leslie has long emphasized, the ventral stream is for representing any stable visual properties as they relate to different downstream cognitive capacities (Kravitz et al., 2013; Ungerleider & Mishkin, 1982), and in previous work (albeit in passing), she acknowledged the tension between the dual stream model and category selectivity, or stimulus-specific “clustering” (Kravitz et al., 2013). Thus, Pitcher and Ungerleider can also be seen as offering a characterization of the STS branch of the ventral stream not in terms of category selectivity but instead based on the downstream goal of social interaction.

Conclusion

When Leslie passed away, we were reminded of a line from W. H. Auden's poem “In Memory of W. B. Yeats” (1940) and reflected on how (to paraphrase) when the current of her feeling fails, the scientist, like the poet, becomes her admirers. As a reflection of our admiration for Leslie, we have attempted to provide a partial answer to the conceptual question “What is a visual stream?” We argued against a simple answer to the question by highlighting ways in which the low-resolution textbook depictions it is based on obscure the underlying complexity of the branching heterarchy that is the visual system. We then tried to build toward a more complex, taxonomic answer that ultimately allowed us to explicate individuation conditions for visual streams, taking inspiration from both the philosophy of science on neural mechanisms and pathways and Leslie's body of work. On the basis of these conditions, we have suggested that the case for a third visual stream has not yet been made, but nonetheless is suggestive of other changes to how we think about the organization of the visual system. In this regard, we hope that Leslie's legacy will flow on through further revisions to the dual stream model of the human and primate visual systems.

Corresponding author: J. Brendan Ritchie, The Laboratory of Brain and Cognition, The National Institute of Mental Health, 10 Center Drive, Bethesda, MD 20814, or via e-mail: john.ritchie@nih.gov.

Author Contributions

J. Brendan Ritchie: Conceptualization; Writing—Original draft; Writing—Review & editing. Sebastian Montesinos: Conceptualization; Writing—Review & editing. Maleah J. Carter: Conceptualization; Writing—Review & editing.

Funding Information

J. B. R., S. M., and M. J. C. were supported by the Intramural Research Program of the National Institute of Mental Health (https://dx.doi.org/10.13039/100000025), grant number: ZIAMH002909 to Chris I. Baker.

Diversity in Citation Practices

Retrospective analysis of the citations in every article published in this journal from 2010 to 2021 reveals a persistent pattern of gender imbalance: Although the proportions of authorship teams (categorized by estimated gender identification of first author/last author) publishing in the Journal of Cognitive Neuroscience (JoCN) during this period were M(an)/M = .407, W(oman)/M = .32, M/W = .115, and W/W = .159, the comparable proportions for the articles that these authorship teams cited were M/M = .549, W/M = .257, M/W = .109, and W/W = .085 (Postle and Fulvio, JoCN, 34:1, pp. 1–3). Consequently, JoCN encourages all authors to consider gender balance explicitly when selecting which articles to cite and gives them the opportunity to report their article's gender citation balance.