Evidence for a third visual pathway specialized for social perception

Abstract

Existing models propose that primate visual cortex is divided into two functionally distinct pathways. The ventral pathway computes the identity of an object; the dorsal pathway computes the location of an object, and the actions related to that object. Despite remaining influential, the two visual pathways model requires revision. Both human and non-human primate studies reveal the existence of a third visual pathway on the lateral brain surface. This third pathway projects from early visual cortex, via motion-selective areas, into the superior temporal sulcus (STS). Studies demonstrating that the STS computes the actions of moving faces and bodies (e.g., expressions, eye-gaze, audio-visual integration, intention, mood) show that the third visual pathway is specialized for the dynamic aspects of social perception.

The two visual pathway model requires revision

Almost forty years ago Ungerleider and Mishkin described a model of primate cortex that proposed two visual pathways [1]. Each pathway was defined anatomically and functionally based on lesion studies in non-human primates. The ventral pathway projects along the ventral brain surface; it computes the identity of visual objects (e.g., faces, animals, cars, tools). The dorsal pathway, also called the vision for action pathway [2], projects along the dorsal brain surface; it computes the location of visual objects and the actions related to those objects (e.g., reaching, grasping, throwing, utilizing). Mapping behavioural functions to visual pathways in this way enabled researchers to build common cognitive frameworks that incorporated findings from different species (human and macaque) and different experimental methods (e.g., neuroanatomy, physiology, neuropsychology, neuroimaging). While recent revisions have incorporated a wealth of subsequent research [3,4], the two pathways model is still principally concerned with the ‘what’, ‘where’ and ‘how’ of visual object recognition. However, current thinking has not kept pace with empirical advances, and it is now clear that the model requires revision.

In the current paper we present evidence for the existence of a third visual pathway on the lateral brain surface (Figure 1, Key Figure). This is consistent with the neuroanatomical and functional inputs that project from early visual cortex, via the motion-selective middle temporal (MT) area, into the STS in both human and non-human primates [5–11]. Regarding the issue of cross-species homologies, the current evidence suggests that only the dorsal bank and fundus of the macaque STS correspond to the human STS. We also review studies demonstrating that motion, and specifically biological motion (e.g., facial and body movement), drives the neural response to visual stimuli in the STS [12–19]. In addition to these responses to visual stimuli, the STS also responds to the human voice [20], language [21] and the audio-visual integration of speech [22]. Based on this evidence it is clear that the third visual pathway is anatomically and functionally distinct from the ventral and dorsal visual pathways. While the ventral and dorsal pathways are concerned with the ‘what’, ‘where’ and ‘how’ of visual object recognition, the third pathway is principally engaged in the dynamic aspects of social perception [23–25].

Figure 1, Key Figure. Cortical connectivity of the third pathway (in red).

The third pathway begins in primary visual cortex (V1) and projects into the posterior banks of the superior temporal sulcus (STS) via the motion-selective area V5/MT. The cortico-cortical connections of the third pathway are independent of the ventral pathway (shown in green) and the dorsal pathway (shown in blue).

Anatomical and functional connectivity of the third pathway

The most compelling evidence for a cortical pathway into the STS that bypasses the ventral pathway comes from non-human primate neuroanatomy. Tracer studies in macaques reveal the existence of a cortico-cortical connection that projects from primary visual cortex (V1) directly into the motion-selective area MT. MT has direct anatomical connections with anterior visual motion areas in the medial superior temporal (MST) and fundus of the superior temporal (FST) cortices [6,7]. FST then feeds into the more anterior regions of the dorsal bank and fundus of the STS (Figure 2). Importantly, this pathway is anatomically segregated from cortico-cortical connections between V1, V2 and V4 that project directly into the inferior temporal cortex of the ventral pathway [26]. Such direct neuroanatomical data are not available in humans, but tractography studies have identified a white matter pathway projecting into the human STS that is anatomically segregated from white matter pathways on the ventral surface [8,27,28]. This convergence of human and non-human evidence demonstrates the existence of a direct pathway into the STS (the dorsal bank and fundus of the STS in macaques) from early visual cortex that is independent of the ventral pathway. The presence of MT demonstrates the fundamental role of motion in the third pathway. Defining the functional properties of the third pathway is therefore dependent on the use of moving visual stimuli.

Figure 2. Cortico-cortical connections of the macaque occipitotemporal cortex.

A direct cortical pathway from primary visual cortex (V1) to the middle temporal (MT) motion processing area is highlighted in red. The third pathway in macaques projects from V1 to MT. MT then has direct anatomical connections with the medial superior temporal (MST) and fundus of the superior temporal (FST) cortices. FST then feeds into the more anterior (rostral) regions of the dorsal bank (d) and fundus (f) of the STS [4].

In non-human primates, moving points of light and basic 3D shapes have been used to map the differences in visual field responses across motion-selective visual areas (MT, MST and FST). Visual field mapping studies (in which visual stimuli are presented in the contralateral and ipsilateral visual fields) establish the functional connectivity of anatomically connected brain areas. For example, in macaques, the parts of visual areas V1, V2 and V4 with dense anatomical interconnections also represent the same parts of the contralateral visual field [29]. Visual field mapping in macaque motion-selective cortex has shown that visual areas that respond to motion in the contralateral visual field progressively represent a greater proportion of the ipsilateral visual field when moving anteriorly along the STS (MT, MST and FST) [5]. These results are also consistent with a study of neurons located in the dorsal bank and fundus of the macaque STS [30]. Results showed the majority of sampled neurons responded to moving more than stationary stimuli and that receptive field sizes encompassed almost the entirety of the visual field. Subsequent neuroimaging studies in humans have shown a similar pattern. The anterior areas of motion processing cortex (hMT+) represent a greater proportion of the ipsilateral visual field than more posterior motion areas [31]. This converging evidence from macaques and humans suggests that lateral brain areas with anatomical inputs into the STS do not exhibit the same contralateral visual field biases observed in ventral visual areas [29,32,33].

To further investigate this putative difference in visual field representations between the ventral and third pathways, we performed a visual mapping neuroimaging study using stimuli designed to maximise the functional response in the STS in human participants [10]. Face videos (with actors posing different facial expressions) were shown in different parts of the visual field. Consistent with prior studies [32,33] we observed a contralateral visual field bias in V5/MT, as well as in two face-selective areas, the fusiform face area (FFA) and occipital face area (OFA) (Figure 3A and 3B). By contrast, we observed no visual field bias in the face-selective area in the posterior STS (pSTS). This difference in visual field bias between the pSTS and OFA was replicated in a subsequent transcranial magnetic stimulation (TMS) experiment. TMS delivered over the right pSTS impaired facial expression discrimination in both visual fields, while TMS delivered over the right OFA impaired facial expression discrimination in the contralateral visual field only. More recently, this same differential pattern between the OFA/FFA and pSTS has been reported in an fMRI mapping study using cartoon faces [28].

Figure 3. The third pathway exhibits no visual field bias.

A. Face-selective areas in the human occipitotemporal cortex. The posterior superior temporal sulcus is in the third pathway. The occipital face area and fusiform face area are both in the ventral pathway. B. There is a contralateral visual field bias in face-selective areas in the ventral pathway (10,11,32,33), while the pSTS had no visual field bias [10]. This same lack of a visual field bias is also seen in the dorsal bank and fundus of the macaque STS [5,30]. C. Group social interactions necessitate directing and redirecting attention to different individuals across the visual field. The lack of visual field bias in the pSTS is consistent with the functional role of the lateral pathway in social cognition.

These results reveal a functional dissociation between face-selective areas in the ventral pathway (FFA and OFA) and the third pathway (pSTS) in human cortex. Specifically, the contralateral visual field bias observed in the ventral pathway is absent in the pSTS. Because the ipsilateral response in the pSTS can only have come from the contralateral hemisphere, there must be a greater degree of interhemispheric connectivity in the third pathway than the ventral pathway. The lack of visual field bias in the STS is also consistent with the type of information decoded from moving faces that supports social cognition. Computing the locations and movements of multiple biological organisms across the entire visual field is essential. Social interaction is commonly conducted with a group of other individuals. When interacting with a group, processing demands will alternate between individuals across hemifields. For example, when someone on your far left raises a hand you redirect your attention from the wider group to the individual. Or, when interacting with only one individual in a group, it can also be necessary to monitor the behaviour of those in the surrounding group (Figure 3C). We argue that the third visual pathway has evolved to compute face information across the entire visual field to support social interaction, which, by its very definition is a dynamic and continually changing process.

The third pathway processes moving faces

The cognitive functions performed in a particular brain area can be deduced (at least partially) by the anatomical connectivity of that area. In the third pathway, the connections between V5/MT and the STS demonstrate the crucial role of motion. Neuroimaging studies show that the face-selective area in the human pSTS [12,24,34] exhibits a greater response to moving faces than static faces [13,14,35]. By contrast, ventral face-selective regions, like the FFA and OFA, show little (or no) preference for dynamic over static faces. An additional face-selective area in the anterior STS (aSTS) [35] also exhibits a greater response to moving than static faces [14,37]. A causal connection between the two STS areas selective for moving faces was demonstrated in our study that combined TMS and functional magnetic resonance imaging (fMRI) [38]. Participants were scanned while viewing face videos after TMS was delivered over the right pSTS. TMS reduced the response to moving faces in the pSTS, aSTS and in face-selective voxels in the amygdala. This result, together with the tractography studies showing an anatomical pathway along the STS [8,27,28], demonstrate the existence of a functional pathway projecting along the STS specialized for moving face perception.

Neuropsychological studies of patients with cortical lesions offer a unique way to causally demonstrate the independent anatomical and functional connectivity of the third pathway in the human brain. The existence of a direct pathway into the STS was proposed as early as 1984 based on the report of prosopagnosic patient GKT, but the lack of structural brain imaging made the theory speculative [39]. Today, functional brain imaging studies have identified multiple prosopagnosic patients who exhibit face-selective responses in the STS, despite having lesions encompassing the brain area in which the FFA and OFA are typically located [40–43]. This causal demonstration that disrupting the ventral pathway has no effect on the response to faces in the STS is consistent with the third pathway being specialized for moving face perception. We tested this hypothesis in an fMRI study of Herschel, a prosopagnosic patient with a right ventral occipitotemporal lesion [11]. The neural response to moving and static faces was measured in face-selective areas and in V5/MT. Results showed the response to moving and static faces in Herschel’s right pSTS and right V5/MT was not significantly different from control participants. There was an impaired response to all faces in Herschel’s right FFA and OFA, which was consistent with his lesion. This differential pattern of activation demonstrates that a neural response to face stimuli in the STS can occur even when face-selective areas in the ventral pathway have been damaged or destroyed. In addition, both Herschel and control participants exhibited no visual field bias for moving faces in the rpSTS. This normal visual field response in Herschel’s rpSTS (despite a reduced contralateral response in his right OFA) further suggests an anatomical segregation between the third and ventral pathways.

This conclusion is seemingly inconsistent with established face processing models. Both cognitive and brain models of face processing [44,45] stipulate that all face information (e.g., the structural form that facilitates identity recognition and the changeable aspects such as eye and mouth movement) is processed via an initial single-entry point. This early structural encoding stage is located in the inferior occipital gyrus, also called the OFA when defined using fMRI [46]. An influential alternative model has also been proposed that suggests moving faces are processed via a pathway that runs from early visual cortex into the STS via V5/MT [47]. This pathway is anatomically and functionally independent of the ventral face processing pathway (for structural face information) that begins in the OFA, before progressing via the FFA into the anterior temporal lobe. To further dissociate the role of motion across face-selective areas, we combined TMS with fMRI to disrupt the two pathways that support face perception in healthy participants [9]. TMS was delivered over the right OFA, or right pSTS while participants were scanned with fMRI viewing moving or static faces. Disruption of the right OFA reduced the neural response to both static and moving faces in the right FFA. By contrast, the response to dynamic and static faces was doubly dissociated in the right pSTS. Namely, disruption of the right OFA reduced the response to static but not moving faces, while disruption of the rpSTS itself reduced the response to moving but not static faces. The dissociation in the response to moving and static faces in the STS, together with the neuropsychological data [11], shows that the STS has cortical inputs that are independent of the OFA. The convergence of causal evidence, the tractography, the selective response for moving faces, and the visual field mapping data show that, in humans, the third pathway has anatomical and functional properties that are distinct from the ventral pathway.

The third pathway in non-human primates

The functional properties of the third pathway have also been defined in non-human primates. These studies, mostly in macaque monkeys, enable researchers to employ invasive experimental methods that are difficult (if not impossible) in human subjects. For example, tracer studies in macaques report a cortical pathway that projects along the STS into the lateral nucleus of the dorsal amygdala [48,49]. Physiology studies have examined neurons in the STS that selectively respond to visual images of faces for over fifty years [50–54]. fMRI studies subsequently identified at least six face-selective patches along the length of the STS [55] and the functions of these areas have been extensively investigated [56–61].

One recent fMRI study has characterised the macaque face patches in a manner consistent with the two-face pathway model [18]. Results demonstrated that face patches located on the dorsal bank of the STS showed a selective response to faces in natural motion, while face patches on the ventral bank responded selectively to the structure of faces (Figure 4A). The face patch MF (located in the mid-fundus of the STS) exhibited a dual response to moving and static faces consistent with this pattern. This functional dissociation of a face-processing pathway that processes structural form, and another that processes changeable facial aspects is consistent with the third vs. ventral face-processing pathway model proposed in humans [9,47,62]. We have also shown that the AF (anterior fundus) face patch preferentially responds to moving faces relative to static faces [37]. More recently we further demonstrated that patches AF (anterior fundus) and MF were most sensitive to changes in facial expression, while patches AL (anterior lateral) and ML (middle lateral) were most sensitive to changes in head orientation [63]. Taken together, these studies begin to demonstrate a functional dissociation between face patches in the macaque STS that is consistent with the anatomical and functional dissociation between moving and static faces in human third and ventral pathways. Namely, face patches in the dorsal bank and fundus of the macaque STS (AF, MD and MF) may correspond with the STS in humans, while those on the ventral bank of the macaque STS (AL and ML) and anteroventral and posteroventral to the STS (AM and PL, respectively) may correspond with face-selective areas in the ventral pathway in humans. This hypothesis is further supported by recent structural data comparing white matter tracts across species [64]. However, this conclusion also raises the question of how we should compare STS face studies across species.

Figure 4. Biological motion processing in the macaque STS.

A. Macaque face patches are organised in a manner consistent with two functionally distinct pathways. Dorsal Patches (purple) AF and MD respond selectively to faces in natural motion. Ventral patches (red) PL, ML and AL respond selectively to static face images. Face patch MF exhibits a split response to moving and static faces consistent with the dorsal / ventral distinction [18]. B. Results from an fMRI study of macaques viewing natural videos that contained animals (left) or no animals (right) [19]. Surface maps show the percent variance explained by biological motion. The brain on the bottom left shows how biological motion drives the neural response from early visual cortex and into the STS.

There is an obvious discrepancy between the human and macaque studies. Face areas in humans are located on both the ventral brain surface (e.g., the FFA) and on the lateral brain surface in the STS. In macaques, by contrast, the face patches studied with fMRI are predominantly located on the brain’s lateral surface. It is not fully clear what accounts for this discrepancy. Recording studies have identified face cells on the ventral brain surface [65,66], but mouth and jaw muscles cause signal drop out that substantially impairs the fMRI data recorded from the ventral brain surface. One study overcame this issue by using an optimised fMRI protocol that was able to identify multiple face-selective areas in the ventral temporal cortex and medial temporal lobe [67]. Despite this finding the majority of macaque fMRI face processing studies have continued to focus on the face patches in the STS. It is clear that future research is required to resolve how face-selective patches identified in macaques should be compared with those identified in humans.

The third pathway processes moving bodies

Faces are not the only moving biological stimulus that are selectively processed in the third pathway. The human STS also responds to visual images of the body [24]. Body-selective responses in STS have been identified in response to point-light walkers [15,68], moving bodies [16] and videos of actors performing observable physical actions [17]. Physiology studies in macaques have also identified cells in the STS that selectively respond to images of bodies and body parts (e.g., hands) [69–73]. Given these findings it is likely that the STS is functionally connected to parietal and frontal areas that compute actions and intentions. For example, visual analysis of goal-directed hand actions in the STS [74] may influence parietal and frontal systems that compute actions, intentions and body movements [75–78].

Despite these findings, the most heavily studied body-selective area in humans is not located in the STS. The extrastriate body area (EBA) [79] is slightly inferior and posterior to the STS, located on the lateral brain surface in Brodmann area 18. This places it in the same brain location as V5/MT, with which it can overlap [80]. Given this overlap between the EBA and V5/MT it is likely that the EBA is located in the third pathway. We recently provided evidence consistent with this hypothesis by demonstrating that the EBA exhibited a greater neural response to moving bodies than to static bodies [81]. A neuropsychological study that reported a normal EBA response to moving bodies in a patient with a lesion to the right ventral occipitotemporal cortex further suggests the EBA is in the third pathway [82]. This hypothesis is seemingly inconsistent with prior neuropsychological patients who were able to perceive biological motion despite reporting visual motion processing impairments [83,84]. However, in the early study there was no functional brain imaging, so the presence of V5/MT was not established [83]. In the later study, while the patient could identify the actions accurately, adding a small number of “noise” dots impaired her performance, suggesting that segregating actions by motion is (at least in part) reliant on V5/MT [84].

fMRI studies have also identified multiple body-selective patches in the macaque [36,52,85–87]. More recently, one study comprehensively demonstrated the extent to which biological motion drives the neural response along the STS [19]. Macaques were scanned using fMRI while viewing videos that showed monkeys interacting in natural environments. Biological motion was the predominant driver of the neural response in clusters that began in early visual cortex and projected along the length of the entire STS (Figure 4B). This result was even more striking in the face-selective patches in the STS, which showed a greater response to biological motion than to faces. This study shows that computing the visual actions of other biological organisms is a fundamental role of the STS. This is consistent with an influential paper that surveyed the face and body responses in the STS and proposed that a full visual representation of the body is necessary to compute the range of socially relevant visual cues [23]. In addition to computing body information relevant for social interaction, the third pathway is also likely to be important for non-social encounters, for example, in encounters with predators and prey.

The third pathway and higher socio-cognitive functions

In this paper we have focused on studies that demonstrate the anatomical and functional connectivity between early visual cortex and the STS via V5/MT. Our goal has been to demonstrate why it is necessary to add the third pathway to the well-established model of dorsal and ventral visual pathways. In this final section we briefly describe evidence demonstrating the higher cognitive functions that are computed in the third pathway. These are primarily engaged in social cognition, namely, understanding and interpreting the actions and behaviors of other biological organisms. The role of the STS, and by extension the third pathway, in social cognition is well established [23–25]. The best evidence comes from the extensive literature demonstrating how the STS responds to a wide variety of social cues. The information used by primates to calculate the meanings and intentions of others is generated by their actions. These actions are generated by faces, bodies, speech and by sound.

Studies of the human STS have shown that it contains regions that selectively respond to exactly these types of visual and auditory stimuli. These include facial expressions and eye gaze [88–93] bodies [16,17], point-light walkers [15,68], the human voice [20], language [21] and the audio-visual integration of speech [22]. In addition, the temporoparietal junction (TPJ) (an adjacent brain area posterior and superior to the STS) responds to theory of mind tasks [94] in which participants are required to interpret the actions of characters in brief stories. One study that simultaneously mapped the responses to multiple types of social input in the STS identified regions that selectively responded to specific stimuli (e.g., faces or voices) as well as regions that responded to multiple contrasts (e.g., language and theory of mind tasks) [95]. This proximity of brain areas computing multi-sensory information relevant to social interactions further dissociates the third pathway from the established roles of the ventral and dorsal pathways.

Concluding remarks

There are many different experimental methods for studying the cognitive functions of primate visual cortex. Tracer studies map the anatomical connectivity between visual areas [5–7]. The neural response to visual stimuli can be identified at different scales using neuroimaging (cortical patches) or physiology (neurons) [12,16,23,24]. Neuropsychological and TMS studies demonstrate which brain areas are necessary for specific tasks and causally define the behavioural functions associated with damaged areas [1,2,11,39,96–98]. We argue that characterizing the primate visual system as visual pathways enables us to describe the cognitive functions of the brain at a level that encompasses all these methods. This creates a common framework that facilitates understanding between those who study the brain at the behavioural, cognitive and neural levels. We have identified some of the unresolved issues and suggestions for future research that will increase this understanding in the Outstanding Questions Box.

Our principle aim in the current paper has been to expand the original two pathways model to include a third pathway. It is clear that while ‘what’, ‘where’ and ‘how’ can describe the many facets of visual object recognition, these terms are wholly inadequate when it comes to describing the complexity and nuances of even basic social interactions. There is no simple one-word description that can encompass the functions of the third visual pathway. Rather, it seems that the visual input into the STS is integrated with other sensory inputs to enable primates to understand and interpret the actions of others.

Outstanding Questions Box.

• Why are macaque face patches mostly identified on the lateral brain surface, while human face patches are found on the lateral and ventral brain surface?

• Do face patches on the ventral bank of the macaque STS show a greater contralateral visual field bias than face patches on the dorsal bank of the STS?

• To what extent is the third pathway reliant on form and structural information computed in the ventral pathway for visual face and body recognition?

• Is the third pathway lateralized to the right hemisphere in humans? If so, what are the visual functions of the left STS and what is the role played by speech?

• What are the anatomical projections from the third pathway to the frontal lobe? What cognitive functions do these projections serve?

• Why does the third visual pathway show no visual field bias while the ventral pathway shows a contralateral visual field bias?

• Is the extrastriate body area (EBA) part of ventral pathway or part or the third pathway?

• Do the low-level motion processing deficits exhibited by autistic individuals impact the typical processing of moving biological stimuli in the third pathway?

Highlights.

• The two visual pathway model of primate visual cortex needs to be updated. We propose the existence of a third visual pathway on the lateral brain surface that is anatomically segregated from the dorsal and ventral pathways.

• The third pathway exists in human and non-human primates. In humans the third pathway projects from early visual cortex into the superior temporal sulcus (STS). In macaques the third pathway projects from early visual cortex into the dorsal bank and fundus of the STS.

• The third pathway has distinct functional properties. It selectively responds to moving faces and bodies. Visual field mapping studies show the third pathway responds to faces across the visual field to a greater extent than the ventral pathway.

• The third pathway computes a range of higher socio-cognitive functions based on dynamic social cues. These include facial expression recognition, eye gaze discrimination, the audiovisual integration of speech and interpreting the actions and behaviors of other biological organisms.