Emotion processing and the amygdala: from a ‘low road’ to ‘many roads’ of evaluating biological significance

Abstract

A subcortical pathway through the superior colliculus and pulvinar to the amygdala is commonly assumed to mediate the non-conscious processing of affective visual stimuli. We review anatomical and physiological data that argue against the notion that such a pathway plays a prominent part in processing affective visual stimuli in humans. Instead, we propose that the primary role of the amygdala in visual processing, like that of the pulvinar, is to coordinate the function of cortical networks during evaluation of the biological significance of affective visual stimuli. Under this revised framework, the cortex has a more important role in emotion processing than is traditionally assumed.

There is tremendous interest in the question of how salient, emotional and socially-charged visual stimuli are processed by the brain. This topic is important because it addresses a fundamental question regarding how biological ‘value’ is assigned by an animal to stimuli in its environment: which stimuli are good and which are bad; which should be approached and which should be avoided. The topic is also intriguing because it fuels questions about modularity in the brain (that is, whether there is a specialized way of, or even dedicated neural substrates for, the processing of affective stimuli). The overarching ‘standard hypothesis’ runs roughly as follows: ecologically important (emotional and social) stimuli are processed initially by a dedicated, modular system that operates rapidly, automatically (without the need to pay attention) and largely independently of conscious awareness¹. Defects in this system are suggested to underlie phobias, mood disorders and post-traumatic stress syndrome, and variability in its functioning reflects individual differences at the genotypic and personality level^2,3.

The hypothesis has two central and related components. The first component is the purported role of the amygdala in the rapid, automatic and non-conscious processing of emotional and social stimuli. The second central component is the proposal of a specific subcortical route of information processing — the so-called ‘low road’ (REF. ⁴) — that bypasses the presumably slower, resource-dependent cortex and that culminates in the amygdala by way of the superior colliculus and the pulvinar nucleus of the thalamus (FIG. 1a). The fact that this pathway bypasses the cortex is thought to imbue the processing of emotion-laden visual stimuli with the above list of properties.

Figure 1. Visual pathways.

a | A traditional flowchart of visual processing typically emphasizes the LGN–V1–V2–V4–TEO–TE pathway, although the scheme is not strictly hierarchical. The amygdala, in particular, is a recipient of visual signals from the anterior visual cortex. According to the ‘standard hypothesis’, a subcortical pathway involving the superior colliculus and the pulvinar nucleus of the thalamus provides fast and automatic access to the amygdala. b | An alternative view of the flow of visual signals includes multiple pathways, including both alternative routes (for example, LGN to MT) and shortcuts (for example, V2 to TEO). Only some of these are shown. The flow of visual information may be more appropriately viewed in terms of ‘multiple waves’ of activation that initiate and refine cell responses at a given processing ‘stage’. For simplicity, feedback pathways, which are known to be quite extensive, have been omitted. The existence of such feedback pathways dictates, however, that a complex ebb-and-flow of activation sculpts the neuronal profile of activation throughout the visual cortex, and likewise the amygdala responses. Some of the connections between the pulvinar and visual cortex, and between the pulvinar and ‘associational’ areas, are also indicated. The line in the pulvinar is intended to schematically separate the medial pulvinar (to the right of the line) from the rest of the structure. FEF, frontal eye field; LGN, lateral geniculate nucleus; MT, medial temporal area (also known as V5); OFC, orbitofrontal cortex; SC, superior colliculus; TE, inferior temporal area TE; TEO, inferior temporal area TEO; V, visual cortex; VLPFC, ventrolateral prefrontal cortex.

Given that the standard hypothesis is shaping both basic and clinical research, this Perspective article provides a critical re-examination of this hypothesis that we hope will re-orient thinking about the processing of emotion stimuli and the roles of the amygdala and pulvinar therein. The main points that we make are as follows: first, there is no evidence for a functional subcortical route for visual processing in primates; second, the cortex plays a larger part in the processing of affective visual information than is typically acknowledged; third, the visual processing of emotion stimuli occurs no faster than visual processing in the cortex in general; fourth, the amygdala’s contribution to processing of affective visual information arises from its broad connectivity with the cortex and other subcortical structures; and finally, the pulvinar plays a part in the processing of emotion stimuli through its extensive connectivity with cortical sites.

The standard hypothesis

The data and theory that underpin the standard hypothesis are not typically articulated in detail, and its central concepts are often vague. The main argument is that insofar as affectively-laden information has survival value, it has driven adaptations in information processing that are reflected in a functionally and structurally modular system². The purported modularity of the system entails automaticity⁵: owing to the potency of affective information, this information is processed independently of attention and awareness. For example, threat-expressing faces have been reported to be processed pre-attentively in visual search paradigms⁶, and fearful faces break into consciousness more quickly than happy expressions during continuous flash suppression (a technique used to render visual stimuli non-conscious)⁷. Moreover, haemodynamic responses in the amygdala have been reported to occur in response to presentation of fearful faces that have (putatively) been rendered invisible by backward masking^8,9 and even during unmasked presentation of fearful faces in patients with blindsight^10,11.

It is also assumed that the anatomical components of the system enable emotion processing to occur to a substantial extent subcortically¹². This suggestion has its roots in rodent studies that demonstrate the existence of a subcortical pathway, through the auditory thalamus to the amygdala, that is sufficient for some forms of auditory Pavlovian fear conditioning⁴. It is assumed that a similar subcortical route exists for visual information processing in primates, including humans (see below). The notion of such a subcortical pathway is appealing because subcortical visual processing is assumed to be faster than cortical visual processing, and processing of affective stimuli is thought to be adaptive in part, because it is fast. For example, judgments of threat can be made from facial stimuli that are presented for as briefly as 39 ms (backward masked)¹³. Because the pathway is assumed to be subcortical, processing of visual information along this pathway is assumed to be coarse. Thus, coarse (that is, low-spatial-frequency) information from affective stimuli is thought to engage subcortical visual processing, consistent with findings that the amygdala (a collection of subcortical nuclei) is activated more strongly by emotional faces presented with low than high spatial frequency¹⁴.

In this Perspective we aim to discuss several shortcomings of the standard hypothesis and to propose an alternative view that we think is better justified by the data, and that we hope will stimulate new research directions. This Perspective does not focus specifically on the amygdala, as several comprehensive reviews about the role of this structure in processing affective information already exist^15,16. Instead, we emphasize physiological and anatomical data concerning the pulvinar, the key ‘link structure’ of the subcortical pathway. Although most of the data described in this Perspective are well known to parts of the neuroscience community, the lack of appreciation for these data by ‘affective science’ researchers could partly explain the widespread acceptance of the standard hypothesis.

We begin by discussing the functional properties of the processing of affective visual information that are central to the standard hypothesis and focus on the issues of speed and coarseness. These issues are relevant to the subsequent discussion of the pulvinar and to the alternative scheme we propose for explaining the properties of visual processing of affective information. Other important notions that are linked to the standard hypothesis, including the modularity of the brain and the roles of attention and awareness in visual processing, are discussed only briefly because they have been reviewed elsewhere^17–21. Finally, we propose an alternative view that assigns a larger role to cortical processing of affective visual information. We suggest that this scheme, which we call the ‘multiple-waves model’, can explain the types of findings that have been used to support the standard hypothesis.

General functional issues

Affective visual information is not processed faster than other visual information

Electrophysiological responses evoked by visual stimuli can be modulated by the emotional content of the stimuli, and this modulation has been reported to occur at short latencies — in some studies in humans, within ~100 ms of stimulus onset^22,23. In addition, the N170 component of the electro encephalography (EEG) signal (or the M170 component in magnetoencephalography (MEG) studies), which is associated with face identification, is in some studies modulated by the emotional expression of the presented face^24,25. However, numerous studies only showed effects with longer latencies, ranging from 200–400 ms (for example, REF. 26). Even in the studies reporting short latency responses to emotional visual stimuli, localizing the neural sources of those responses using EEG or MEG is problematic and, therefore, the origin of the signals in these studies might not be in the subcortical pathway.

In addition, single-unit recordings in monkeys indicate that responses in the cortex (even in the frontal cortex) occur with latencies that are within the range of the latencies observed in subcortical areas (BOX 1). This is consistent with the idea of a rapid, feedforward ‘cortical sweep’ of visual information processing. Moreover, behavioural and electrophysiological studies of object processing and perception in humans suggest that visual processing in general (that is, including non-affective processing) can be remarkably fast and that a substantial amount of information can be gathered from even a single glance at a natural scene (see Supplementary information S1 (box)). For example, a recent study showed that only 19–67 ms were required to attain 75% correct performance on several tasks, including determining a scene’s global property (for example, ‘natural scene’) and basic level categorization (for example, ‘forest’)²⁷. Thus, there is nothing particularly special about the processing speed of affective information. Finally, the speed of cortical — as opposed to subcortical — visual processing could also account for the reported rapid modulation of evoked brain responses by stimulus valence. Valence, like other affectively relevant stimulus dimensions, is probably computed in several brain regions, one of which is likely to be the orbitofrontal cortex (OFC). Short-latency (100–150 ms) electrophysiological responses in the OFC have been associated with discrimination of the valence of a visual stimulus²⁸. We develop this final point in greater depth below, in the context of our new proposal.

Box 1 | Speed of visual processing

One way to assess the speed of visual processing is to measure and contrast response latencies across brain areas. For example, do responses in the purported subcortical pathway occur earlier than those in cortical sites? The figure shows that in the macaque cerebral cortex, the earliest latencies are remarkably short, and even mean response latencies indicate remarkably fast cortical processing⁷². Areas that became active at the given latency after visual stimulation are shown in red, those that were activated earlier in yellow and those that were not yet activated in white. Areas for which no information was available are shown in dark grey (see the figure). Visual response latencies in the pulvinar are between 60–80 ms and overlap with latencies observed in early visual cortical areas V1 and V2 (REF. ¹⁰⁴). In the inferotemporal cortex (that is, ‘late’ visual cortex) latencies can be as short as 60–85 ms⁷² and, strikingly, in some frontal sites such as the frontal eye fields (FEF) as short as 40–70 ms. These latencies again overlap with those in area V1 (REFS ^84,105). Thus, although mean response latencies increase gradually from posterior to anterior visual cortices, there is considerable overlap (see the figure). In the context of the standard hypothesis, it therefore seems that pulvinar responses are not particularly fast. However, it is of interest that visual response latencies in the superior colliculus are somewhat faster than those observed in the pulvinar, showing an early, transient response around 40–70 ms that may support rapid eye movements during orienting¹⁰⁶ (note that these response times overlap with FEF responses).

What are the response latencies of neurons in the amygdala? In the monkey amygdala, responses to visual stimuli range from 100–200 ms^30,107–109, although shorter response latencies to unspecific stimuli (for example, fixation spots) have been reported³⁰. Differences in evoked responses between threatening and neutral or appeasing facial expressions in the monkey amygdala have been found in the range of 120–250 ms³⁰. Intracranial studies in humans generally find the earliest single-unit responses to visual stimuli around 200 ms^33,110. Moreover, in one study, modulation of amygdala responses by the affective content of stimuli was observed to start at 200 ms²⁶ (see also REF. 110).

In summary, subcortical visual processing is not discernably faster than cortical processing. Furthermore, the crucial variable is not the timing of the initial stimulus responses but the time at which reliable differences between affective and non-affective stimuli can be detected. It has been suggested¹¹¹ that most of the information encoded by visual neurons may be available in 100-ms-long segments of activity (that is, spiking data within a 100 ms epoch) and that a fair amount of information is available in segments of 50 ms, and even some of 20–30 ms (note that these segments consider post-latency neuronal spikes only). Although these data demonstrate the remarkable speed of neuronal computation (at least under some conditions), they add milliseconds to the time that is required to, for example, discriminate between stimuli. A final consideration is that responses in humans are possibly slower than in monkeys. For example, in one study in humans, the fastest recording sites had response latencies of just under 60 ms and were probably located in V1 (or possibly V2)¹¹². In the monkey, the fastest responses in V1 can be observed under 40 ms⁷². 5, Brodmann area 5; 7a, Brodmann area 7; 7ip, Brodmann area 7ip (intraparietal); 8a, Brodmann area 8a; EC, entorhinal cortex; FEF, frontal eye field; FST, fundus of superior temporal cortex; IPa, superior temporal area IPa; M1, primary motor cortex; MST, medial superior temporal cortex; MT, medial temporal area (also known as V5); OFC, orbitofrontal cortex; PFC, prefrontal cortex; PGa, superior temporal area PGa; PreM, premotor cortex; SEF, supplementary eye field; SMA, supplementary motor area; TAa, anterior subregion of superior temporal area TA; TE1, inferior temporal area TE1; TE2, inferior temporal area TE2; TE3, inferior temporal area TE3; TEm/TEa, medial and anterior subregions of inferior temporal area TE; TPO, superior temporal area TPO; TS, superior temporal sulcus. Figure is reproduced, with permission, from REF. 72 © (2000) Cell Press.

Taken together, the findings above indicate that processing of affective visual stimuli is no faster than cortical processing of visual stimuli in general. Indeed, cortical visual processing is both efficient and fast. Thus, the argument that a separate subcortical system is required for fast perception of affective stimuli is problematic (Supplementary information S1 (box)).

Processing of affective visual stimuli involves both coarse- and high-spatial-frequency information

According to the standard hypothesis, the subcortical pathway is particularly effective at carrying low-spatial-frequency information, mainly because the superior colliculus and pulvinar are assumed not to convey much high-spatial-frequency information. This notion was initially based on findings in rodents that simple (‘coarse’) auditory conditioning does not require the cortex, whereas conditioning that requires more complex stimulus discriminations does⁴.

Neuroimaging studies in humans seem to be consistent with this idea. For example, in one study amygdala responses were stronger when participants viewed low- compared with high-spatial frequency fearful faces¹⁴, and when they viewed fearful faces versus neutral faces both at low spatial frequency¹⁴. Similarly, activation in brain areas consistent with the location of the superior colliculus and pulvinar was greater in response to fearful faces than to neutral faces at low spatial frequencies¹⁴. Findings of this kind have been interpreted as suggesting that the amygdala is relatively ‘blind’ to high-spatial-frequency information. However, the amygdala receives major projections from the anterior inferotemporal cortex²⁹ that convey highly processed object information — in fact, the amygdala receives highly processed cortical input from all sensory modalities except olfaction²⁹. Indeed, electrophysiological studies have shown that the monkey amygdala contains neurons that are tuned to the identity of specific faces^30,31 and that the human amygdala shows category-specific responses (for example, for animals or natural scenes)^32,33. These are properties that require high-spatial-frequency information.

Moreover, it has been shown that the discrimination of facial expressions relies on both low- and high-spatial-frequency information³⁴. The perception of fear is particularly reliant on high-spatial-frequency information³⁵. Indeed, a study in a patient with bilateral amygdala lesions showed that this patient’s impaired recognition of facial expressions of fear was due to impaired processing of the eye region of faces, especially of high-spatial-frequency information about the eyes³⁶. These results demonstrate the importance of high-spatial-frequency information in fear recognition and indicate that the amygdala is required for this type of visual processing.

In summary, although some findings are consistent with the notion that subcortical areas process coarse visual information, the perception of emotional expressions actually involves both coarse and fine information. Furthermore, the amygdala not only receives inputs that convey fine spatial information but seems to be crucially involved in using this information to decode facial expressions (Supplementary information S1 (box)).

The amygdala is not essential for rapid, non-conscious detection of affective information

The standard hypothesis often draws on additional themes, such as the roles of attention and awareness in visual processing, and the more general issue of the extent to which emotion and cognition are processed by separate circuits in the brain. Both of these issues are complex and have been discussed at greater length elsewhere^19,20, but we briefly comment on attention and awareness.

Processing affective information is known to occur under some conditions of inattention and unawareness. However, as discussed elsewhere, the interpretation of the published literature in terms of ‘strong automaticity’ is unwarranted²⁰. Briefly, behavioural, functional MRI (fMRI) and EEG studies indicate that when processing resources are sufficiently consumed (for example, by engaging attention on challenging tasks), visual processing of emotional stimuli is greatly reduced or eliminated. This challenges the notion that emotional stimuli are processed automatically (see REFS ^37–39 for further discussion of the role of attention). A recent study of a patient with complete amygdala lesions⁴⁰ perhaps provides the most decisive data on this issue. In this patient, reaction times for detecting fearful faces among distractor stimuli were within the normal range, and fearful facial expressions broke into consciousness faster than happy faces during binocular suppression to the same degree as in control subjects (FIG. 2). These findings demonstrate that the amygdala is not essential for non-conscious, rapid fear detection, at least in the tasks used in this study⁴⁰ (see also REF. 41).

Figure 2. Intact non-conscious processing of fearful faces in the absence of the amygdala.

In a subject with complete amygdala lesions (subject S.M.), fearful faces broke into consciousness during continuous flash suppression with latencies similar to those of control subjects. a | Experimental stimulus: fearful or happy faces were shown to the non-dominant eye while a flashing Mondrian pattern was shown to the dominant eye. This technique is called continuous flash suppression as it suppresses the visibility of the stimulus presented to the non-dominant eye. The contrast of the Mondrian pattern was gradually decreased while that of the face was increased until subjects could detect the face and indicate it with a button press to establish reaction time. b | Plots of reaction times (RT) in the task. In the case of S.M., fearful faces broke interocular suppression faster than happy faces (shown by the red bar) and to the same degree as 7 demographically matched healthy controls (shown by blue bars; the mean and standard deviation are also shown). Figure is reproduced, with permission, from REF. 40 © (2009) Macmillan Publishers Ltd. All rights reserved.

Thus, independent lines of evidence challenge the notion that processing of affective visual information occurs independently of attention and awareness. Moreover, some of the properties typically connected in the literature with automaticity (for example, detecting fearful faces among distractors) may not entirely depend on the amygdala.

Physiological and anatomical issues

The pulvinar is a key link element in the purported colliculus–pulvinar–amygdala pathway. Here, we briefly review physiological and anatomical data regarding the pulvinar — and the subcortical pathway of which it is thought to be a component — that are relevant in the context of the standard hypothesis. In particular, we discuss data that are relevant to the question of whether this structure is better conceptualized as a relatively passive way station or as a dynamic element of brain circuitry.

Pulvinar input

The pulvinar complex is the largest nuclear mass in the primate thalamus and is thought to have expanded in size during evolution in parallel with other visual structures⁴². The pulvinar does not seem to exist in brains of rodents and other small mammals⁴³. In terms of connectivity that is relevant for visual processing, it receives direct visual input from the retina, indirect visual input via the superficial layers of the superior colliculus and massive input from striate and extrastriate visual cortices (FIG. 3). All of these projections terminate in the inferior pulvinar. Intriguingly, however, the visual response properties of pulvinar cells do not reflect those of neurons in the superior colliculus, and the precise contribution that input from the superior colliculus makes to pulvinar responses remains uncertain⁴³. For example, superior colliculus lesions have little effect on electrophysiological responses of pulvinar neurons, in contrast to striate cortex lesions, which abolish responses in the inferior pulvinar⁴⁴. In a related fashion, collicular and pulvinar lesions result in different behavioural impairments⁴⁵ (see also REF. 46). These observations argue that the pulvinar may be better thought of as participating in cortical networks, rather than as relaying visual information from the superior colliculus. This is borne out by the finding that, unlike the lateral geniculate nucleus (LGN), the pulvinar’s driving inputs in fact originate in the cortex, whereas subcortical inputs to the pulvinar are typically modulatory^47,48; for this reason, the pulvinar is described as a higher-order thalamic nucleus, as opposed to a first-order nucleus such as the LGN⁴⁹.

Figure 3. Schematic layout of the pulvinar.

Some traditional characterizations of the pulvinar emphasize the inferior (Inf), lateral (Lat) and medial (Med) nuclei. Most pulvinar nuclei (including other nuclei and sub-nuclei that are not shown here) are involved in thalamo–cortical loops that target different cortical territories (shown in blue)⁴²,⁶⁰. The inferior nucleus is reciprocally connected to striate and extrastriate cortices, the lateral nucleus is connected to association cortices in temporal and parietal lobes (although it is also interconnected with the extrastriate cortex) and the medial nucleus is connected to the higher-order association cortex in parietal, frontal, orbital (not shown), cingulate and insular regions (the insula is not shown), in addition to the amygdala. Thus, the medial nucleus, which is of great interest in the present context, is not only connected with the amygdala but is also part of multiple thalamo–cortical loops (note, however, that the connection to the amygdala does not seem to be bidirectional). The superior colliculus is a layered structure whose superficial layers are visual in nature and project to the inferior nucleus. Its intermediate and deeper layers are multimodal and involved in motor preparation, including for eye movements, and project to the medial nucleus. A ventrolateral to dorsomedial axis that is helpful in understanding the organization of pulvinar nuclei and potential ‘ventral’ and ‘dorsal’ domains is shown by a dotted line (see also REF. 42 for a related scheme). Figure is modified, with permission, from REF. 43 © (2004) CRC Press. IT, inferior temporal cortex; MT, medial temporal area (also known as V5).

Pulvinar activity

Studies in monkeys and humans with pulvinar lesions have suggested that this structure is involved in determining what is salient in a visual scene^50,51. Consistent with this notion, the response of pulvinar neurons to visual stimuli is increased if attention is paid to the stimulus or if the stimulus has behavioural relevance. For example, pulvinar neurons respond more vigorously to behaviourally relevant targets than to unattended stimuli⁴⁵. In one study, as many as 92% of pulvinar cells exhibited attenuated responses to stimuli that were task-irrelevant (that is, passively viewed)⁵² compared to stimuli that were task-relevant. Furthermore, the impact of attention on evoked responses in the pulvinar is spatially specific, such that a pulvinar neuron only increases activity when a monkey attends to a stimulus that falls within the receptive field of the cell⁵³. Finally, the pulvinar seems to be crucial (as was shown in a study in which the pulvinar was pharmacologically inactivated) when a distractor stimulus has to be ‘filtered out’⁵⁴. Thus, it has been proposed that the pulvinar is involved in attention and/or distractor filtering, and this is consistent with data from neuroimaging and lesion studies in humans (Supplementary information S1 (box)).

The pulvinar is also important for visual awareness. For example, lesion studies in humans have revealed that pulvinar damage is associated with visual neglect and with feature-binding deficits^51,55,56. A recent study in monkeys is particularly noteworthy: here, neural activity was recorded in the pulvinar during a visual illusion that induced the intermittent perceptual suppression of a bright luminance patch⁵⁷. Neurons in the pulvinar showed changes in spiking rate according to trial-by-trial stimulus visibility, suggesting that they reflected the visual awareness of the stimulus. Similarly, a recent fMRI study in humans found that the pulvinar responded not to the affective significance of visual stimuli but to whether or not they were consciously perceived⁵⁸, again in a trial-by-trial manner. The fMRI results are consistent with a study in which pulvinar responses were associated with a subject’s percept of a change⁵⁹. Notably, responses were observed during ‘false alarm’ trials (those in which a stimulus change was reported but did not actually occur) but not during ‘miss’ trials (those in which a stimulus change occurred but went unnoticed by the participant). These results do not support the suggestion that the pulvinar is involved during non-conscious processing, and are inconsistent with a major role for this structure in the subcortical pathway proposed by the ‘standard hypothesis’, according to which the pulvinar behaves as a relatively passive relay.

Pulvinar anatomy

It is also important to consider the anatomical features of the pulvinar that highlight its extensive bidirectional connectivity with the cortex. For example, all 20–30 known visual areas connect with the pulvinar, sometimes in a relatively topographic fashion^43,60. Temporal, parietal, cingulate, frontal and insular cortices are all connected with the pulvinar as well. At a gross level, it is as if the entire convoluted cortex were ‘shrink-wrapped’ around the pulvinar⁶⁰. Based on the connectivity data, one can discern a ventrolateral to dorsomedial axis, and this has led to the suggestion that the pulvinar may contain two connectional ‘domains’ (REFS ^42,60) (FIG. 3). The ventral domain (containing the inferior pulvinar) is densely connected with the visual cortex (including areas V1–V4 and MT)⁶⁰. It therefore has a strong visual component (that is, it could be called the ‘visual pulvinar’), and its projections to the dorsal visual stream may mediate some of the visual abilities in people with blindsight^61,62 (Supplementary information S1 (box)). The dorsal domain (including the traditional medial subdivision; see below) has connections with the cross-modal association cortex, including temporal and parietal areas (including area 7A and the lateral intraparietal (LIP) cortex — areas that are also involved in attention)⁶⁰. The dorsal domain receives highly processed visual input from the inferior temporal gyrus area TE (and some from area TEO) in the inferotemporal cortex⁶⁰. It is also connected with the cingulate cortex, frontal cortex (including the OFC), insula and amygdala⁶⁰ (see below). The dorsal domain is therefore much more ‘associational’ and, in fact, has remarkable potential to integrate information from very diverse brain regions (FIG. 3).

In summary, the pulvinar is a complex structure with important visual and integrative properties. Functional studies have characterized several ways in which the pulvinar is modulated by attention and awareness. Indeed, the pulvinar is likely to be an important ‘control site’ for attentional mechanisms more broadly⁶³. Taken together, these data are antithetical to the standard hypothesis, which assumes that automatic processing is mediated by a subcortical pathway involving the pulvinar.

Does the subcortical pathway exist in primates?

Work on fear conditioning has shown that there are direct, subcortical projections to the amygdala from the auditory thalamus (that is, from the medial geniculate nucleus (MGN)) in rats. An analogous projection carrying visual information from the visual thalamic nucleus (that is, the LGN) to the amygdala has not been documented in rodents or primates. However, it is frequently assumed that a colliculus—pulvinar—amygdala pathway exists in the case of vision. Here, we review relevant data from primates that question this assumption.

Anatomical studies in monkeys have reported connections between the superior colliculus and the pulvinar^42,43, and between the pulvinar and the amygdala^64,65. As described above, the superficial superior colliculus projects to the inferior pulvinar — both of these structures can be considered to be ‘visual’. Yet, the inferior pulvinar is extensively interconnected with the visual cortex (consistent with visual functions) but not with the amygdala. Instead, the projection to the amygdala originates in the medial pulvinar^64,65 (although the strength of this connection may be relatively weak⁶⁶), a nucleus that is extensively interconnected with much of the cortex, as described above. Furthermore, like other thalamic nuclei, the primate pulvinar is thought to have neither excitatory nor inhibitory long-range intrinsic connections⁶⁷. Considered together, these results do not support the idea of a colliculus–pulvinar–amygdala visual pathway.

To summarize, there is no evidence for a direct or an indirect subcortical pathway conveying visual information to the amygdala in monkeys. It is therefore unclear how findings from auditory fear conditioning studies in rodents can be applied to visual processing of affective stimuli in primates (see also BOX 2). For further discussion of the pathway from intermediate and deep layers of the superior colliculus (which are multimodal in nature and linked to eye movements) to the pulvinar see Supplementary information S1 (box).

Box 2 | Subcortical processing: audition in rats versus vision in primates

Historically, the standard hypothesis has derived a considerable portion of its motivation from the organization of the auditory system in rodents. However, the auditory and visual systems differ in important ways. The temporal precision of the auditory system is substantially greater than that of the visual system. In contrast to vision, audition is omnidirectional, such that information from all directions can be sampled (though at relatively low spatial resolution). Furthermore, the functional anatomy of the auditory system is very different from that of the visual system. Properties such as sensitivity to sound frequency, duration, amplitude, pitch and binaural disparity are already observed at subcortical levels. In fact, the primary auditory cortex (A1) seems to be involved in high-level functions and is therefore not equivalent to ‘visual cortex transplanted into the auditory modality’. Indeed, there are several subcortical stages below the level of the auditory cortex, and it has been suggested¹¹³ that the role of the inferior colliculus in auditory processing might be equivalent to that of primary visual cortex (V1) in vision and that the A1 is more analogous to visual areas in the inferotemporal cortex than to V1.

These considerations suggest that a subcortical pathway for auditory input to the amygdala (in rodents) would not actually be analogous to the purported subcortical visual pathway (in monkeys). Accordingly, auditory connections from the medial geniculate nucleus in the thalamus to the amygdala, although bypassing cortex, already convey highly processed information, in contrast to the suggested monkey (visual) counterpart.

The multiple-waves model

The standard hypothesis has influenced both basic and applied research and at first glance has intuitive appeal. However, we have shown that, in its current form, the hypothesis is problematic in multiple respects. We therefore suggest that a revision of the standard hypothesis is in order.

Multiple visual pathways and coarse information processing

One of the primary motivations for the standard hypothesis is the perceived need for rapid processing: fast but coarse visual processing is just what an organism needs in a dangerous environment. We argue that visual pathways other than a colliculus–pulvinar–amygdala pathway carry out this role (see also REFS ^68,69).

Visual processing along the ventral processing stream, which is crucial for object recognition, has historically been described to occur in a relatively hierarchical fashion. However, important ‘short-cut’ connections link areas V1 to V4 (REF. ⁷⁰), V2 to TEO⁷⁰ and V4 to TE⁷¹ (FIG. 1b), providing the means for faster information transmission to the inferotemporal cortex⁷². Direct connections between the LGN and extrastriate regions, including V2 (REFS ^73,74) and V4 (REF. ⁷³), have also been reported. Indeed, combined electrophysiology and fMRI studies in monkeys have shown robust visual activation in areas V2 and V3 in animals with lesions of V1, demonstrating that routes bypassing V1 can be sufficiently potent to drive extrastriate visual responses⁷⁵ (see REF. 76 for evidence in humans). A combined lesion and fMRI study in monkeys⁷⁷ revealed widespread extrastriate activation in the absence of V1 and demonstrated a role of LGN-dependent projections for visual detection. V1-independent responses were observed in areas V2, V3, V4, MT (also known as V5), the fundus of the superior temporal cortex (FST) and the LIP area. These findings establish the importance of the LGN for at least some types of blindsight. Longer range ‘short-cuts’ also exist, such as those that link regions in the ventral visual cortex with the ventrolateral prefrontal cortex (including the OFC)⁷⁸. It has been proposed⁷⁹ that low-spatial-frequency information may rapidly reach parietal and frontal cortices from the early visual cortex⁸⁰, thereby providing coarse information about the gist of a visual scene and supporting object recognition⁸¹. It is thus possible that these initial ‘volleys of activation’ are less susceptible to manipulations of attention and awareness, especially given that they may primarily convey information from the magnocellular system^80,82. Furthermore, computational models that assume a purely hierarchical structure of the visual system have failed to provide a good fit to the existing latency data⁸³, which is consistent with the existence of bypass connections.

Thus, there are multiple parallel routes for visual information processing that lead to substantial temporal dispersion of evoked responses and that enable ‘high-level’ regions to respond with surprisingly short latencies⁸⁴. Each processing stage adds approximately 10 ms to the latency⁸⁴. The ‘cost’ of using such bypassing stages may be that, at first, only relatively coarse information is available about a visual item. This is consistent with a coarse-to-fine processing strategy in which the more-global contents of a stimulus are processed earlier than finer details^85,86.

Based on the considerations above, we suggest that the initial processing of visual information proceeds simultaneously along parallel channels, creating ‘multiple waves’ of activation across the visual cortex and beyond⁸⁷. In this manner, visual stimuli that have affective and motivational significance can engage multiple brain sites — including the amygdala, OFC, anterior insula and anterior cingulate cortex — that can direct processing towards these behaviourally relevant items. Hence, rapid processing of affective information is possible even in the absence of a specialized subcortical pathway (FIG. 1b) or a single specific structure such as the amygdala⁴⁰.

In light of our proposal, we suggest that affective blindsight involves some of the alternate pathways described here. A recent EEG study of a patient with complete cortical blindness used advanced source modelling to investigate the time course of information processing⁸⁸. Although all facial expressions, including neutral ones, evoked relatively short-latency responses (70–120 ms) localized to the superior temporal sulcus, emotion-specific responses that were localized to the anterior temporal cortex, and possibly the amygdala, occurred considerably later (120 and 200 ms later, respectively). Although this study suffers from the localization problems alluded to above, the findings are consistent with the notion that affective significance is computed in parallel along several circuits (see also REF.89 for a related proposal).

In light of the change of focus from a single, specialized subcortical pathway to a multiple-pathway model, it is important to reconsider the roles of both the pulvinar and the amygdala during processing of emotional visual stimuli.

The role of the pulvinar in processing of emotional visual stimuli

Connections between the pulvinar and amygdala have been reported^64,65, suggesting that the pulvinar may have a role in emotion processing. As mentioned above, it has been proposed that the pulvinar is involved in determining the behavioural relevance of a stimulus, directing attention to a stimulus and determining awareness of a stimulus. Based on the data described above, we propose that the pulvinar helps to coordinate and/or regulate the flow of multimodal information via a series of thalamocortical loops (FIG. 3). This proposal takes into account that most of the input to the pulvinar comes from the cortex.

In the context of emotion processing, the most relevant nucleus of the pulvinar is probably the medial nucleus, given that it connects not only with the amygdala but also with a larger array of other brain regions. We therefore suggest that it may be involved in more general functions that impact emotion processing, such as determining the behavioural relevance and/or value of a stimulus. For example, the medial nucleus is connected with parietal regions that are involved in attention. It is also connected with the OFC and cingulate cortex, which are important for computing an object’s biological value. Furthermore, it is connected with the insula, a region that has a role in emotional feelings. These connections are all bidirectional (except the connection with the amygdala), providing opportunities for modulating and regulating information flow. According to our proposal, the importance of the pulvinar in emotion is not due to its status as a subcortical ‘labelled line’ conveying emotional information to the amygdala, but due instead to its pattern of connectivity with subcortical and cortical sites that have a role in determining the biological significance of a stimulus.

Studies by Ward and colleagues have investigated the impact of pulvinar lesions on processing of affective visual information in humans. A complete unilateral loss of the pulvinar led to a severe deficit in a patient’s ability to recognize fearful expressions shown in the contralesional visual field⁹⁰. Within the framework suggested here, when weak and/or brief visual stimuli have biological significance, cortico–pulvino–cortical circuits coordinate and amplify signals in a manner that enhances their behavioural impact. This framework is consistent with the impairment in recognizing fear in patients with pulvinar lesions and also with their impairment in recognizing anger (and possibly happiness)⁹⁰. Notably, the essential pulvinar damage was found in the medial pulvinar, the region that projects to the amygdala in monkeys. The proposed framework is also consistent with a study that reported that viewing complex unpleasant images impaired performance in a subsequent simple (neutral) visual task in control subjects, but not in a patient with pulvinar damage⁹¹ — according to our framework, the unpleasant stimulus did not garner additional resources in the patient (which would have interfered with performance, as it did in the controls).

Pulvinar involvement in the processing of affective information does not seem to reflect emotion per se, however. In an fMRI study in humans⁵⁸, a simple contrast between affective and neutral conditions did not reveal different responses in the pulvinar. Instead, there was a significant relationship between the magnitude of evoked responses in the pulvinar and the probability of correctly detecting a target on a trial-by-trial basis during the affective condition but not during the neutral condition (FIG. 4a). These results reveal an emotion-by-visibility interaction that may characterize the role of the pulvinar more generally. In other words, we suggest that the pulvinar amplifies responses to stimuli of potential value to the animal (such as one that signals the possibility of shock in the experiment) (FIG. 4b).

Figure 4. Pulvinar and amygdala during processing of affective stimuli.

a | Logistic regression analysis of evoked responses in the left pulvinar as a function of affective significance for a sample individual during an attentional blink task⁵⁸. The slope of the logistic fit indicates the strength of the predictive effect. For clarity, only binned data for the conditioned stimulus (CS+) condition are included (shown by orange dots). The grey line shows the fit for these data, and the blue line shows the fit for data from the neutral stimulus (CS−) condition. The inset shows mean logistic slopes across individuals, revealing that a relationship was detected for the affective (CS+) but not the neutral (CS−) condition. b | The medial pulvinar is proposed to amplify evoked responses of behaviourally-relevant stimuli via circuits involving the cingulate cortex, orbitofrontal cortex (OFC) and amygdala, all regions important for the valuation of an incoming stimulus. c | Valuation signals in the amygdala affect behaviour by impacting responses across the brain. During an attentional blink task using affective stimuli, a response in the amygdala to a stimulus predicted that the stimulus would be detected⁹⁶. Statistical path analysis revealed that this effect is mediated through projections from the amygdala to the visual cortex, as well as through projections involving the prefrontal cortex (PFC). fMRI, functional MRI. Data in part a from REF. 58.

The role of the amygdala in processing affective visual stimuli

What part is left for the amygdala to play in the processing of affective visual stimuli? Its connectivity pattern provides some clues. The predominant source of visual input to the amygdala, specifically the basolateral nucleus, comes from higher-order visual association cortices in the anterior temporal lobe²⁹. This suggests that the amygdala is a convergence zone for highly-processed sensory information that is relevant to object processing. In addition, there are loops between the visual cortex and the lateral and basal nuclei of the amygdala, and this feedback is thought to modulate visual processing⁹². Further integrative functions of the amygdala stem from its extensive connections with much of the cortex. In addition to its well-recognized connections with medial and orbital territories of the prefrontal cortex, the amygdala is also connected to the lateral prefrontal cortex, albeit in a weaker manner⁹³. Importantly, the architecture of the prefrontal cortex (PFC) is such that, on average, inputs from the amygdala reach approximately 90% of the PFC after a single connection within the frontal cortex⁹⁴. Furthermore, the amygdala seems to be part of a ‘core brain circuit’ (REF. ⁹⁵) that is topologically central in terms of global brain connectivity and whose functions could include aggregation and distribution of information.

In light of these considerations, we propose that the amygdala’s contribution to the processing of affective visual information arises not from a subcortical source of visual input, but from its broad connectivity with the cortex and other subcortical structures. Given this connectivity, the impact of the amygdala on behaviour can be mediated through many routes, for example, via both the visual cortex and prefrontal cortex. This suggestion is consistent with findings of a study that combined the attentional blink task with fear conditioning⁹⁶. For emotion-laden stimuli, trial-by-trial fluctuations in evoked responses in the amygdala predicted whether or not a target was detected. Furthermore, this impact of the amygdala on behaviour was mediated by both the visual cortex and prefrontal cortex (as suggested by statistical path analysis), consistent with the idea that during the processing of affectively significant items, the amygdala enhances sensory processing through both direct (amygdala–visual cortex) and indirect (amygdala–prefrontal cortex–visual cortex) paths (FIG. 4c).

For reasons of brevity we have discussed the amygdala as a single entity, but it should be noted that the amygdala is in fact a complex structure comprised of more than a dozen nuclei. In particular, the central nucleus has extensive descending connections to the hypothalamus and other brainstem nuclei that regulate autonomic and endocrine responses and, in this manner, contributes to several aspects of emotional expression and mobilization of bodily resources. Among others, this is an important distinction between the roles of the amygdala and pulvinar during processing of affective visual stimuli.

Conclusions

The evidence we have reviewed here suggests that the idea of a subcortical pathway that is specialized for the processing of emotional stimuli should be revised. Our reinterpretation has important implications for the conceptualization of the amygdala’s function in the processing of emotional visual information. We suggest two revised roles for the amygdala. First, the amygdala has a mostly modulatory role in a wide array of networks. The precise functional importance of the amygdala in these networks remains to be investigated, but it is unlikely that it will map specifically onto emotion. Instead, we think that it corresponds to broader and more abstract dimensions of information processing, including processing of salience, significance, ambiguity, unpredictability^21,97–99 and other aspects of ‘biological value’. More broadly, we argue that the amygdala has a key role in solving the following problem¹⁰⁰: how can a limited-capacity information processing system that receives a constant stream of diverse inputs selectively process those inputs that are the most relevant to the goals of the animal¹⁰¹?

Thus, the amygdala serves to allocate processing resources to stimuli, at least in part by modulating (through its connectivity) the anatomical components that are required to prioritize particular features of information processing in a given situation. Such a role would come into play not only for affectively significant stimuli but also for other stimuli. Notably, the amygdala may not be unique in this respect as there are other, largely parallel, networks with architectures that do not include the amygdala but that also enable diverse functions — notably the network subserved by the connections between the cortex and the pulvinar. We think that our proposal is consistent with the majority of findings and can accommodate several views of amygdala function^16,102,103, with the difference that it provides a broader and more flexible perspective.

The second aspect of our proposed revision is to stress the speed and temporal dispersion of cortical processing, which render moot the assumed need for a fast subcortical route. Many visual properties can be processed very rapidly by the initial wave of cortical response, which suggests that there is ample time for substantial feedback, even within the cortex. Consequently, understanding the flow of visual information within the cortex should help us to understand how affective stimuli are processed. Ultimately, the fate of a biologically-relevant stimulus should not be understood in terms of a ‘low road’ versus a ‘high road’, but in terms of the ‘multiple roads’ that lead to the expression of observed behaviours.

There is an enormous literature implicating the amygdala in affective dysfunction in nearly every psychiatric illness, most notably in mood disorders. The revised view in this Perspective suggests that rather than focusing on neurons within the amygdala, we should focus on connections within the cortex and between the cortex and subcortical structures such as the amygdala. This may not come as a big surprise to some readers as, in the main, it simply reflects the idea that the substrate of brain function is not so much to be found within neurons as within networks.

Glossary

Attentional blink

A phenomenon that occurs in experiments in which a rapid stream of visual items is presented to an observer whose task is to detect two targets within the stream. When the two targets are separated in time by a brief interval (for example, 200–500 ms), the successful detection of the first target impairs detection of the second one (as if the participant blinked) owing to limited processing capacity.

Backward masking

A phenomenon that occurs in experimental paradigms in which a target visual stimulus is followed by another salient visual stimulus that ‘masks’ the perception of the target stimulus, making its detection or recognition difficult or impossible. Visual masking is commonly used to manipulate visual awareness.

Blindsight

The ability, in humans or monkeys, to respond to visual stimuli without consciously perceiving them — a situation that may ensue following a lesion to the primary visual cortex.

Continuous flash suppression

A technique in which a fixed image shown to one eye is suppressed by a stream of rapidly changing images flashed to the other eye. The technique is used to manipulate visual awareness.

Labelled line

A processing architecture in which a separate pathway conveys information that is specific to a class of sensory stimuli owing to, for example, receptor specificity (for example, pain and touch conveyed by particular somatosensory channels).

Magnocellular system

A visual pathway from the retina to the cortex that conveys relatively fast, transient and wavelength-insensitive information.

Path analysis

A statistical method to investigate the relationship between multiple variables.

Source modelling

A set of techniques that attempt to estimate the neural ‘sources’ of the electrical or magnetic signals that are measured at external sensors (for example, at the scalp in the case of electroencephalography).

Visual search

An experimental paradigm in which subjects are asked to indicate the presence or absence of a ‘target’ item (for example, a fearful face) among an array of ‘distractor’ items (for example, neutral faces).