Modulation of visual processing by attention and emotion: windows on causal interactions between human brain regions

Abstract

Visual processing is not determined solely by retinal inputs. Attentional modulation can arise when the internal attentional state (current task) of the observer alters visual processing of the same stimuli. This can influence visual cortex, boosting neural responses to an attended stimulus. Emotional modulation can also arise, when affective properties (emotional significance) of stimuli, rather than their strictly visual properties, influence processing. This too can boost responses in visual cortex, as for fear-associated stimuli. Both attentional and emotional modulation of visual processing may reflect distant influences upon visual cortex, exerted by brain structures outside the visual system per se. Hence, these modulations may provide windows onto causal interactions between distant but interconnected brain regions. We review recent evidence, noting both similarities and differences between attentional and emotional modulation. Both can affect visual cortex, but can reflect influences from different regions, such as fronto-parietal circuits versus the amygdala. Recent work on this has developed new approaches for studying causal influences between human brain regions that may be useful in other cognitive domains. The new methods include application of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) measures in brain-damaged patients to study distant functional impacts of their focal lesions, and use of transcranial magnetic stimulation concurrently with fMRI or EEG in the normal brain. Cognitive neuroscience is now moving beyond considering the putative functions of particular brain regions, as if each operated in isolation, to consider, instead, how distinct brain regions (such as visual cortex, parietal or frontal regions, or amygdala) may mutually influence each other in a causal manner.

1. Introduction

Prior to the recent advent of functional neuroimaging, the traditional approach for attributing particular cognitive functions to particular structures in the human brain (often referred to as ‘functional localization’), was to relate specific types of brain damage (originally studied post-mortem but now also with high-resolution structural MRI, e.g. Rorden & Karnath 2004) to patterns of selective cognitive deficit (Damasio & Damasio 1989; Farah 1990; Grüsser & Landis 1991; Shallice 1988). Putative functions were thereby tentatively assigned to particular brain regions, as for Broca's and Wernicke's celebrated ascription of speech production or speech reception to distinct regions of left frontal or temporal cortex. Analogously, early neuroimaging studies with positron emission tomography (PET) and then functional magnetic resonance imaging (fMRI) also sought to assign particular functions to particular brain areas (Petersen et al. 1988; Kanwisher et al. 1997; Epstein et al. 1999). Some commentators critiqued such work for merely confirming or refining what was already known from neuropsychology, now via neuroimaging. But when such confirmation or refinement did arise (which was not always the case!), this provided a useful step in validating the new methods. Moreover, we think that neuroimaging has now moved well beyond mere confirmation or refinement of neuropsychology, with the development of advanced neuroimaging tools. These include use of multiple converging tests to probe operations and representations for specific brain areas (e.g. Tong et al. 2000); use of well-defined cognitive methods for studying internal representations, as for those derived from the psychological repetition priming literature (Grill-Spector & Malach 2001; Naccache & Dehaene 2001; Vuilleumier et al. 2002b); the advent of higher resolution fMRI (Grill-Spector et al. 2006); and development of sophisticated multivariate statistical analyses for assessing patterns of activity within particular brain areas (Norman et al. 2006; see also Rees 2007).

Another traditional perspective in neuropsychology and behavioural neurology, beyond strict ‘functional localization’, also finds an echo in recent functional neuroimaging work. Clinicians have often argued on the basis of lesion evidence from brain-damaged patients that complex cognitive functions, such as memory, attention or language, are not each reliant on any single brain area but instead reflect a ‘distributed network’ of contributing areas (e.g. Mesulam 1990) and cognitive components (e.g. Shallice 1988). Cognitive neuropsychology led to numerous multicomponent models of particular domains in cognition (e.g. Bruce & Young 1986; Hinton & Shallice 1991). Functional neuroimaging has contributed further to the identification and refinement of cognitive networks. For instance, recent fMRI studies have identified brain networks associated with different aspects of attention, including not only superior and inferior circuits in fronto-parietal cortex (Corbetta & Shulman 2002), but also circuits in medial regions of the frontal and parietal lobes (Raichle et al. 2001). As another example, recent fMRI work (along with neuropsychological patient studies) has emphasized not only that some regions of ventral visual cortex may be particularly involved in processing faces (e.g. the so-called fusiform face area (FFA); Kanwisher et al. 1997), but also that an extensive ensemble of further areas contribute to face processing also (e.g. superior temporal sulcus, amygdala, retrosplenial cortex and the putative occipital face area; Haxby et al. 2000; Vuilleumier & Pourtois 2007). Results from human neuroimaging have thus enriched traditional neuropsychological approaches from both ‘functional localization’ and ‘distributed network’ perspectives.

In this article we emphasize that, in combination with other methods, human functional neuroimaging is now embarking on a new critical next step. Rather than merely characterizing the possible functions of particular brain areas, or identifying a distributed network associated with a particular domain of cognition, many current studies now seek to identify how remote but interconnected regions within a particular network may influence each other causally. This combines (and thus goes beyond) both the functional localization perspective and the distributed network perspective, by seeking to determine the functional contributions of a given area to activity taking place in other interconnected regions of the network. Such issues are sometimes referred to as concerning ‘functional integration’ (e.g. Friston 1998, 2002) rather than functional localization per se.

Some pioneering work has already used various forms of ‘effective connectivity’ analyses of imaging data (from PET or fMRI, but also electroencephalography, EEG or magnetoencephalogram, MEG) to illustrate the importance of dynamic interactions between brain areas (Friston et al. 2003; Penny et al. 2004; Valdes-Sosa et al. 2005), and their possible modulation by state- or context-dependent influences (Coull et al. 1999; Buchel & Friston 2000). These approaches typically involve sophisticated mathematical models of neuroimaging data that inevitably must rely on various assumptions when searching for putatively causal influences between brain areas. The time has now come to address causal interactions between human brain regions more directly also, by combining neuroimaging measures (e.g. fMRI, EEG, MEG) with ‘interventional’ manipulations, such as transcranial magnetic stimulation (TMS), pharmacology, or focal lesions in neurological patients. In addition, other causal interventions are possible in animal studies (e.g. transient inactivation of a given brain area by cooling, or highly local drug application, selective deafferentation by tractotomy, genetic manipulation, viral transfection, etc.).

Here, we will illustrate new approaches in human studies that are non-invasive, yet still introduce a causal dimension. One approach applies neuroimaging in patients with focal brain lesions to uncover the functional effect of damage to one particular area upon neural activity in other surviving areas. Pharmacological fMRI is also a growing area of research for assessing causal effects of interventions in human studies (e.g. Bentley et al. 2003; Wise & Tracey 2006). Finally, it is now possible to target specific human brain regions online with TMS, actually during fMRI or PET scanning (or EEG), while assessing the causal impact on activity in remote but interconnected regions (e.g. Bestmann et al. 2005, 2006; Paus 2005; Ruff et al. 2006).

2. Attentional and emotional modulation of visual processing: paradigm cases of causal interplay between different brain regions?

There are many aspects of human cognition (perhaps all) for which causal influences between remote brain areas may apply. Here, we illustrate the issue of causal interplay between different brain regions for two specific situations: attentional and emotional modulation of visual processing. In both domains, remote influences between brain regions are implicated in influencing perceptual processing and awareness. In both cases, our understanding of such network interactions has improved through the combination of functional neuroimaging with more interventional techniques. We argue that both attentional and emotional effects on visual perception concern influences upon visual cortex from brain regions beyond visual cortex. In this sense, the effects are analogous. But the critical brain regions and pathways producing causal influences upon visual cortex may be different for the two cases, in accord with the different psychological factors involved. Thus, we will present evidence that the amygdala (among other regions) contributes to emotional modulation of visual processing; while the frontal eye fields (FEF), among other regions, including some parietal areas, contribute to attentional modulation. While there are many previous suggestions of such possible influences from the amygdala or FEF in the literature, here we will concentrate on how true influences of one region upon others might be directly demonstrated for the human brain.

What we mean by ‘modulation of visual processing’ can be illustrated not only by human neuroimaging, as below, but also by direct single-cell recording studies in awake behaving monkeys. Figure 1a shows an example for attentional modulation of visual responses (neural firing rates) in inferior temporal visual cortex (adapted from Chelazzi et al. 1998). Similarly, figure 1b shows an example of emotional modulation of visual processing, also from neurons in temporal visual cortex (adapted from Sugase et al. 1999). In the attentional case (figure 1a), firing of neurons with particular visual preferences is enhanced and prolonged for a given display when the preferred stimulus within it is attended as task-relevant (corresponding to the current item to be searched for by the animal), relative to when that preferred stimulus is task-irrelevant (and a non-preferred stimulus is attended, instead, for the same visual display). In the emotional case (figure 1b), the response to a stimulus that matches the neuron's visual preference (here a preference for face images) is enhanced and prolonged when the seen face conveys an emotional rather than neutral expression. Note that in both cases (figure 1a,b), the initial rise in firing rate seems comparable regardless of attentional or emotional status, with the impact of the latter factors acting to boost and sustain neuronal activity from approximately 100 ms into the neural response.

Figure 1.

Single-cell recordings in the monkey illustrate two different types of modulation of visual responses. (a) Illustration of attentional effects for neurons in inferior temporal visual cortex (adapted from Chelazzi et al. 1998). When the ‘preferred’ stimulus is presented, responses are enhanced and prolonged if attention is directed to the stimulus as task-relevant (red), but weaker and more transient if attention is directed to another stimulus instead, with the preferred stimulus now being task-irrelevant (green). In this case, later components of the response can then be similar to when the preferred stimulus is absent (blue). (b) Illustration of emotional effects on face-selective neurons in superior temporal sulcus (adapted from Sugase et al. 1999). The pattern and time-course of emotional effects appear rather analogous to those of attention, with enhanced and prolonged responses when the seen face has an emotional expression (red), but weaker with a more transient peak when the seen face is neutral (green).

Both of these effects have tentatively been attributed to ‘top-down’ or ‘feedback’ influences from other areas (specifically, from fronto-parietal regions in the case of attentional modulation, and from limbic regions such as the amygdala for emotional effects). However, no direct evidence for this could be derived from studies (such as Chelazzi et al. 1998; or Sugase et al. 1999) that recorded only from the affected visual region. Moreover, it is not inconceivable that both examples could, in principle, be considered as reflecting the same general ‘attentional’ phenomenon. For instance, one might argue that an emotional face becomes more attended than a neutral face, possibly even due to recruitment of fronto-parietal attention circuits. Hence, decisive experiments are required to determine the causal origins of such visual modulations.

Below we turn to human neuroimaging and neuropsychological studies of attentional or emotional effects on visual processing. These effects in humans have several analogies to the monkey single-cell phenomena illustrated in figure 1. While providing some overview of the literature, we focus particularly on studies that may show direct causal influences of remote brain areas upon visual cortex, and that introduce new methods for doing so.

3. Selective attention and modulation of sensory processing

Findings from both human and animal neuroscience provide abundant evidence that top-down modulations of sensory processing play a key role in selective attention and perceptual awareness (Driver et al. 2003; Kanwisher & Wojciulik 2000, for reviews). Across a wide range of paradigms it has been shown that, while holding stimulus displays constant, directing attention towards or away from a particular stimulus (by making it task-relevant or irrelevant) will affect sensory neural processing for that stimulus. This is observed when recording individual neurons and/or local field potentials in awake behaving monkeys (e.g. Gottlieb 2002), and for fMRI, EEG or MEG measures in humans (e.g. Kastner & Ungerleider 2000). Moreover, these attentional modulations of sensory processing can have strong corresponding effects on perceptual judgements and awareness (e.g. Cameron et al. 2002). Typically, sensory responses or activations are enhanced for a given stimulus when attended, relative to the same stimulus when ignored. Such effects have now been found with human fMRI for all areas of retinotopic visual cortex (Hopfinger et al. 2000; Kastner & Ungerleider 2000), including V1 (Ghandi et al. 1999), and even for human lateral geniculate nucleus (O'Connor et al. 2002). Attentional modulations have also been found for feature-selective responses in visual areas (e.g. when attending to colour versus motion, Corbetta et al. 1990; Chawla et al. 1999) and for particular directions of motion (Saenz et al. 2002); together with differential responses to particular stimulus categories, such as visual words versus objects (Rees et al. 1999) or faces versus houses in the FFA and parahippocampal place area (PPA), respectively (Wojciulik et al. 1998; O'Craven et al. 1999). Figure 2 illustrates that the FFA shows a stronger response to displays including face stimuli when those faces are attended (for a perceptual comparison task) rather than ignored (with the comparison task now performed on concurrent house stimuli instead), even though retinal stimulation remains the same in both conditions.

Figure 2.

Example of fMRI results showing modulation of fusiform cortex responses to faces by spatial attention in the human brain (adapted from Vuilleumier et al. 2001a). Subjects saw displays that always contained a pair of faces (aligned either vertically or horizontally), together with a pair of houses, but concentrated on one pair only to perform a picture-matching task. The lateral face-selective fusiform area (FFA) was more strongly activated when faces appeared at the task-relevant location (red), while responses were strongly reduced when faces were task-irrelevant (green), although the visual displays were physically comparable in both conditions (and equated by counterbalancing). In comparison, FFA was not responsive to pictures of houses (blue), obtained in a separate localizer fMRI scan. Units for fMRI activation (betas) correspond to parameters estimates for event-related changes in BOLD signal.

Likewise, in EEG studies, sensory components such as P1 and N1 potentials are typically found to exhibit a greater amplitude for attended relative to unattended stimuli (e.g. Heinze et al. 1994; Martinez et al. 2001). Taken together, all these data indicate that selective attention acts on sensory processing by enhancing neural responses to task-relevant stimuli, with corresponding enhancements of perceptual awareness, whereas unattended information evokes a reduced response or in some extreme cases no differential response (e.g. Rees et al. 1999).

4. Possible sources of attentional modulation

As described above, there are now many clear demonstrations that top-down effects of attention (i.e. of task-relevance) on visual processing can apply at many ‘sites’ within the visual system, ranging from early retinotopic areas through to higher level ventral and dorsal regions. This has led to the new question of which neural system(s) may impose such modulations upon visual pathways, providing the causal ‘sources’ for attentional modulation. A common suggestion is that a broad network of frontal–parietal regions contributes to this, collectively instantiating attentional control in ‘attentional network(s)’ (Mesulam 1999; Driver & Frackowiak 2001; Kastner & Ungerleider 2001; Corbetta & Shulman 2002) that modulate sensory processing in relation to current task demands, by providing top-down signals or biases that influence visual cortex.

Several different types of evidence implicate frontal–parietal areas in such attentional control of sensory processing. Here, we consider how directly causal such evidence may be, noting that in many cases the current evidence is often more correlational than strictly causal or interventionist. Numerous recent neuroimaging studies have highlighted the role of a putative large-scale ‘attention network’ (or networks), involving several areas in parietal and frontal cortex (figure 3), often with some right-hemisphere predominance, typically activated when comparing conditions that require attention shifting to those that do not (e.g. Corbetta et al. 2000; Hopfinger et al. 2000), or conditions requiring active judgements of sensory inputs versus those posing less attentional demand with the same stimulation (e.g. Pinsk et al. 2004; Schwartz et al. 2005). Strikingly, a common network is often found across rather different tasks, whenever visual processing requires more selective attention (figure 3a,b).

Figure 3.

Examples showing activation of attentional-control networks within frontal and parietal cortex in two different visual tasks. (a) Blood oxygen level-dependent (BOLD) fMRI activity is increased in bilateral intraparietal sulcus (IPS), middle frontal gyrus (possibly corresponding to the frontal eye field, FEF) and anterior cingulate cortex (ACC), when subjects have to focus their attention at fixation for a difficult visual task (rapid sequential visual presentation of targets among a stream of distractors, with high attention load), relative to an easy task at fixation with the same stimuli (low attentional load). Adapted from Schwartz et al. (2005). (b) A similar and overlapping network is activated during a visual search task, with increased activation when subjects have to detect a novel target (with different colour and different location) relative to a repeated target (with same colour but different location). Adapted from Kristjansson et al. (2006).

Usually many different areas are activated by such comparisons, but some subsets might be associated with distinct roles. Recent neuroimaging work on attentional control has sought to make increasingly subtle comparisons, leading to various suggestions of distinct subsystems within attentional control. For instance, there may be more ‘anterior’ and ‘posterior’ attention systems (Posner & Dehaene 1994), possibly with more ‘executive’ functions being associated with the former (see also Burgess et al. 2007; Stuss & Alexander 2007). More ‘dorsal’ and ‘ventral’ attention networks have also been suggested (Corbetta & Shulman 2002), possibly with the former involved in directing attention according to the current task set, and the latter involved in interrupting this or serving as a ‘circuit-breaker’ when required. Other putative distinctions between spatial and non-spatial attention (Husain & Rorden 2003), between selective attention versus alerting/arousal (Robertson 1999), or between supramodal versus modality-specific attention (Macaluso & Driver 2001) have been made for various frontal and parietal circuits.

However, a limitation of such neuroimaging findings arises here. When considered as a single method, in isolation from other approaches, neuroimaging is essentially a correlative approach. This can make it hard to assess whether certain activations are essential for performing a particular cognitive function, or merely associated with it, perhaps epiphenomenally. Moreover, merely demonstrating activation of a putative attention network across frontal and parietal cortex, in a situation where attentional modulation of visual cortex also arises (e.g. Kastner & Ungerleider 2000), cannot in itself establish that the observed modulation of visual cortex was causally imposed by particular regions within the fronto-parietal network. Some work using effective connectivity analyses of fMRI data has provided initial evidence in support of dynamic interactions between areas (Buchel & Friston 2000; Friston et al. 2003). But such approaches rely on several assumptions or simplifications in their mathematical models and as yet may fall short of demonstrating strict causality. Below we consider how such causal issues might be addressed more directly.

5. Implication of frontal and parietal cortex in attentional control

Some longstanding evidence that frontal and parietal regions might be involved in attentional effects upon visual perception and visual awareness arises from brain-damaged patients. Many neurological and neuropsychological reports (Heilman et al. 1970; Heilman & Valenstein 1979; Mesulam 1981, 1999; Damasio et al. 1987) concern patients with lesions in (often large) frontal and/or parietal regions (possibly also involving superior temporal cortex, Karnath et al. 2001). These patients can manifest symptoms and deficits that appear to reflect deficits in attention and awareness for incoming stimuli. For instance, in the intriguing ‘spatial neglect’ syndrome (Driver & Vuilleumier 2001; Karnath et al. 2003; Vallar et al. 2003; Driver et al. 2004), patients with extensive unilateral lesions in perisylvian regions (usually on the right side) can appear oblivious to information towards the contralesional side of space, even when they have no primary sensory or motor deficit for that side (e.g. still have intact visual fields). Areas commonly damaged in neglect patients (figure 4) often overlap with those activated in fMRI studies of attentional control in healthy subjects, although some disputes continue about how and why the typical lesion and activation sites may relate. Lesions in some parts of this network may lead to functional disruptions in surviving areas elsewhere in the network (Corbetta & Shulman 2002; Corbetta et al. 2005).

Figure 4.

Common lesion sites in patients with unilateral spatial neglect. (a) Damage may involve different regions in both parietal and frontal lobes, most often the inferior parietal lobule (IPL) and temporo-parietal junction (TPJ), but also the middle frontal gyrus (MFG) and inferior frontal gyrus (IFG) and possibly the superior temporal gyrus (STG). These regions overlap with many of those areas associated with attentional control by fMRI studies in normals (cf. figure 3). (b) Lesions may have very different extents in different neglect patients, as shown here for two example cases (adapted from Driver & Vuilleumier 2001), often involving more than just one brain area within the attentional network.

Contralesional deficits in patients with spatial neglect are often exacerbated by the presence of concurrent competing stimuli on the ipsilesional side, as observed for the phenomenon of perceptual ‘extinction’ during double simultaneous stimulation. Patients may correctly perceive a stimulus presented alone in their left visual field, but fail to detect the same stimulus when paired with another simultaneous event in the right visual field (Bender & Teuber 1946; Heilman et al. 1970). Such deficits have long been considered to involve pathological biases and/or limited capacities in attention, leading to a contralesional stimulus escaping awareness only when it must compete for attentional resources with an ipsilesional event (Heilman & Valenstein 1979; Mesulam 1981; Driver & Vuilleumier 2001; Geeraerts et al. 2005). Space constraints preclude a comprehensive review of neglect or extinction here (Driver & Vuilleumier 2001), but we can briefly emphasize that in manifesting pathological losses in perceptual awareness, such patients illustrate that regions well beyond visual cortex (i.e. parietal, frontal and possibly superior temporal areas, all remote from posterior and ventral visual areas) can make some critical contributions to visual attention and awareness, leading to pathological losses when damaged.

Such lesions beyond the conventional visual system may have functional effects upon vision precisely because they disrupt top-down or recursive causal influences from those regions upon visual cortex (Driver & Vuilleumier 2001; Driver et al. (2001); Kastner & Ungerleider 2001; Corbetta et al. 2005). But alternative possibilities have not always been ruled out. For instance, the mere observation that parietal and/or frontal lesions can lead to pathological losses in visual awareness might, on its own, be consistent with disruption to purely feedforward processes, which normally operate when occipital areas project to higher regions, but inevitably fail at subsequent processing stages when higher regions are lesioned. However, neural processing is increasingly regarded as highly recursive, rather than involving only one-way traffic (Bullier et al. 2001; Pascual-Leone & Walsh 2001). Such issues can be addressed by assessing directly whether focal lesions (or transient disruption) of ‘higher-level’ regions (e.g. in parietal or frontal cortex) lead causally to functional changes in the visual responses of ‘lower’ intact occipital regions, and whether this may relate to the pathological losses in visual awareness in patients.

There have been relatively few such studies to date for neglect and extinction patients. It is only fairly recently that non-invasive measures of neural activity, such as EEG, SPECT or fMRI, have been applied to study visual responses in patients with such deficits after lesions centred on right inferior parietal cortex (though for some pioneering attempts with EEG, see Lhermitte et al. 1985; Spinelli et al. 1994; Viggiano et al. 1995). Several fMRI studies have now demonstrated, via event-related responses to pictures of faces and/or houses, that some residual visual activations can still be found for extinguished stimuli in the contralesional (left) visual field of these patients. Such activations were found not only in right striate and extrastriate cortex (figure 5), but also in some category-selective regions, such as the FFA (Rees et al. 2000, 2002; Vuilleumier et al. 2001b, 2002a). These activations typically became significantly greater and extended into higher-order areas, when the same stimuli were consciously seen by the patients rather than extinguished, as determined by their perceptual report (Vuilleumier et al. 2001b; Rees et al. 2002). The greater activations in visual cortex were also associated with concomitant activations of (and functional coupling with) parietal and frontal regions in the intact left hemisphere (Vuilleumier et al. 2001b). Similarly, event-related potential (ERP) recordings in such patients have also revealed some residual but reduced evoked potentials from visual cortex for visual stimuli extinguished due to parietal lesions (Marzi et al. 2000; Driver et al. 2001; Vuilleumier et al. 2001b), as opposed to when consciously seen by the same patients. Thus, while these studies have revealed unconscious responses to neglected or extinguished stimuli in parietal patients, they have also demonstrated that some visual activations may be reduced by lesions outside visual areas and that such neural changes may relate to impaired conscious perception.

Figure 5.

Effects of right parietal damage on visual cortex activation in a patient with left spatial neglect and visual extinction. Face stimuli were presented in the contralesional left hemifield with another distractor shape in the ipsilesional right hemifield, such that the faces were either perceived on some trials or extinguished from awareness on other trials (adapted from Vuilleumier et al. 2001a). When perceived, contralesional faces evoked significant BOLD fMRI responses in intact right occipital and temporal visual areas (red, here for a region of inferior temporal cortex posterior to FFA). Residual activation was still observed when contralesional faces were extinguished (green), relative to when there was no stimulus in the contralesional field (blue), but such activation was reduced relative to perceived faces (red). These data indicate that parietal damage may have significant functional consequences on the activation of intact visual areas in relation to conscious perception. Units for fMRI activation (betas) correspond to parameters estimates for event-related changes in BOLD signal.

More recent work from our group (Vuilleumier et al. 2004a) used fMRI retinotopic-mapping procedures (Sereno et al. 1995) in right-parietal patients with left neglect and extinction to characterize the responsivity of intact visual cortex and any effects of attentional demand on these areas in more detail. We mapped cortical areas V1 through to V4 for each hemisphere and recorded their fMRI responses to peripheral flickering checkerboards in either visual field. In the main experiment (which was analogous to a study of normal visual attention by Schwartz et al. 2005), the patients had to fixate a stream of successive central stimuli to perform one of the two tasks with varying attentional demand (low or high load; see Lavie et al. 2004; Schwartz et al. 2005), while ignoring peripheral checkerboard stimuli. In normal observers, increasing attentional load at fixation reduces visual activations for the (task-irrelevant) peripheral visual checkerboard, but does so symmetrically (i.e. similar effect of attentional load on both peripheral visual fields). Our fMRI results in right-parietal patients showed a striking pattern. Whereas, visual responses of V1–V4 appeared normal under low attentional load in these patients, even for the left visual field, increasing attentional load at fixation produced a significant asymmetry in visual activation to peripheral stimuli across all successive stages of the visual system. This was found from V1 onwards, but with the largest effect in higher visual areas (Vuilleumier et al. 2004b; Schwartz et al. 2005), such that right V4 no longer responded to a left checkerboard when foveal attentional load was high. These results suggest that early retinotopic cortex for the disrupted side may respond normally under low attentional demands at fixation, but pathologically when attention is engaged by other stimuli. This may lead to corresponding effects on visual awareness (Walker et al. 1991). Indeed, we have now confirmed behaviourally (in collaboration with Jason Mattingley & Chris Rorden) that increased attentional load at fixation can disrupt visual reports for items in the peripheral left visual field more than the right visual field in neglect patients (Lavie & Robertson 2001). More importantly for present purposes, these new retinotopic fMRI data in right-parietal patients indicate that disruptions of visual perception due to lesions in attention-related brain areas may not merely reflect disrupted feedforward access to the (lesioned) higher-level representations. Instead they can involve changes in visual processing within early occipital areas: changes that are functionally caused by the parietal damage and that depend on attentional state.

Frontal lesions in humans can also produce some functional changes in processing within visual cortex, although relatively few studies have directly addressed this. Some reports (Barcelo et al. 2000; Gehring & Knight 2002) indicate that while lesions to dorsolateral prefrontal cortex (DLPFC) do not usually produce persistent signs of the neglect syndrome, they can produce failures to detect visual targets (e.g. ‘oddballs’) in the contralesional hemifield during monitoring tasks. One study applied visual ERP measures to frontal-lesioned patients (Barcelo et al. 2000). Patients were shown rapid visual streams of bilateral stimuli, concurrently in both visual fields. Those with DLPFC damage showed a reduction in relatively early visual ERPs (starting with the P1 component thought to arise from extrastriate cortex), specifically for visual targets in the field contralesional to their DLPFC injury. Such abnormalities extended from approximately 120 ms after stimulus onset for a more sustained period lasting approximately 500 ms, and correlated with behavioural deficits in target detection. These findings indicate that prefrontal cortex may also impose some regulatory influences upon neural activity in extrastriate visual cortex, and that such influences may be necessary for normal visual target detection (Yago et al. 2004).

Taken together, these fMRI and EEG studies of patients with parietal or frontal lesions, focusing on the impact upon processing in visual cortex, illustrate that combining the lesion approach with neural measures of activity (for intact regions) may reveal truly causal influences of parietal or frontal regions upon visual cortex, in relation to attentional or task-related manipulations. Any differences between frontal and parietal influences still remain largely unknown, so further studies should implement comparable tasks in different lesion groups. Related studies are now being conducted in non-human primates (e.g. Orban et al. 2006).

6. Fronto-parietal influences on attentional control in the normal brain: combining brain-stimulation with measures of remote neural activity

Interventional approaches to studying causal influences between remote but interconnected brain regions may not need to rely on lesions only. Another approach is to use ‘neuro-disruption’ techniques (Chambers & Mattingley 2005) to manipulate activity in one brain region, while recording from other interconnected regions. An elegant example comes from recent work by Moore and colleagues (Moore et al. 1998; Moore & Armstrong 2003; Moore & Fallah 2004), who applied microstimulations to the macaque FEF. Single-cell recording work indicates that the FEF are involved not only in saccadic behaviour, but also in selective visual processing, particularly in relation to target–non-target distinctions in search tasks (Bichot et al. 2001; Schall 2004). Moreover, in humans, the FEF are often activated (figure 3) as part of the putative attention network (Kastner & Ungerleider 2000; Corbetta & Shulman 2002), and are also implicated in the lesions of some neglect patients (Mesulam 1999).

Moore and colleagues (Moore et al. 1998; Moore & Armstrong 2003; Moore & Fallah 2004) took advantage of the spatial precision with which saccades can be triggered by FEF microstimulation in monkeys. In this way, they could map out the ‘motor field’ for each of their particular stimulation sites within FEF. Critically, they then stimulated at the mapped site, but now below the threshold level needed to trigger a saccade, while studying the impact of this on visual performance by the monkey during a target-detection task (Moore et al. 1998), or on responses to visual stimuli in V4 neurons with receptive fields that either did or did not correspond spatially with the motor field of the FEF microstimulation (Moore & Armstrong 2003). The striking finding was that subthreshold FEF microstimulation could enhance both visual performance by the monkey and also visual responses in V4, provided there was a spatial correspondence between the visual target (and V4 receptive field) with the motor field of the stimulated FEF site. This provides a direct causal demonstration that (stimulated) activity in FEF can produce spatially corresponding modulations of visual processing in extrastriate cortex, with a corresponding impact on visual performance also.

Microstimulation with implanted electrodes will rarely be available in human subjects (though see Blanke et al. 2000; Zumsteg et al. 2006). However, TMS provides a non-invasive method for causally manipulating neural activity at a targeted site, whose effects are increasingly well studied and understood (Pascual-Leone et al. 2000; Pascual-Leone & Walsh 2001; Paus 2005). Moreover, it is now possible to combine application of TMS to human cortical sites while concurrently recording brain activity with PET, fMRI or EEG, although this can be technically challenging, especially for concurrent fMRI (Bestmann et al. 2005, 2006; Ruff et al. 2006). Ruff et al. (2006) recently applied TMS to human FEF while recording fMRI activity from visual cortex and retinotopically mapping areas V1 through V4. Increased intensity of FEF–TMS led to enhanced activation of peripheral visual field representations, and relative suppression of central visual field representations, for all retinotopic areas of visual cortex, including V1 (figure 6). TMS applied to a control site (vertex) had no such effects. Thus, circuits originating in the human FEF can causally modulate activity in visual cortex, as previously suggested (see above) on the basis of much less direct, less causal evidence. Ruff and colleagues also found that psychophysical visual judgements were modulated by FEF–TMS, in a manner that accorded with their fMRI findings for early visual cortex. Under high intensity FEF–TMS, peripheral visual stimuli were judged as higher in contrast relative to foveal visual stimuli, analogously to the enhanced fMRI responses for peripheral visual field in the visual cortex caused by FEF–TMS.

Figure 6.

Illustration of concurrent TMS–fMRI study by Ruff et al. (2006), with example results from retinotopic visual cortex. TMS was applied over human right frontal-eye fields (see cartoon at left, depicting TMS stimulator held over this brain region). TMS stimulation was applied inside the MR scanner, interleaved with MR slice-acquisition, with procedures to prevent MR artefacts. Example fMRI results are shown for two participants, in ‘flat-map’ depictions of retinotopically mapped visual cortical areas. Borders between adjacent areas (e.g. V1 with ventral V2 (‘V2v’), or with dorsal V2 (‘V2d’) and so on) are drawn in black, with areas labelled. Foveal confluence is marked with a cross, with increased retinal eccentricity running out from here within each marked visual area. The ‘hot’ colours correspond to increased fMRI activity with higher TMS intensity to FEF; ‘cold’ colours represent decreased activity instead. The consistent pattern in all subjects was that, for all retinotopic areas (V1–V4), representations of the peripheral visual field showed enhanced fMRI activity with increased FEF-TMS intensity, while the central visual field (nearer the foveal cross) showed reduced activity. This confirms that human FEF can causally modulate activity in retinotopic visual cortex. Ruff et al. (2006) also derived and confirmed the psychophysical prediction, based on these fMRI data, that TMS should enhance peripheral relative to central vision, for perceived contrast.

In another recent study applying TMS to human FEF, Taylor et al. (2007) recorded ERPs during a task requiring direction of visual attention to the expected side of an upcoming visual target. TMS over FEF (but not over a control site, posterior to motor cortex) modulated ERPs recorded from occipital electrodes over visual cortex, both prior to and during the visual stimulus presentation. Fuggetta et al. (2006) also recently combined TMS with ERP recordings during a visual-attention task (in their case, visual search). They found that the early phase of the N2pc component, often associated with focusing of visual attention (Luck 1995), was eliminated over the right-hemisphere by TMS to right posterior parietal cortex.

Taken together, the recent studies that combine TMS to frontal or parietal sites, together with neural measures of activity in or over occipital cortex, illustrate the potential fruitfulness of a new causal approach that manipulates particular candidate attentional-control regions with neuro-disruption techniques, while studying the impact on remote but interconnected regions of visual cortex, and any corresponding impacts upon visual performance and awareness.

In summary thus far, many studies show that visual processing and neural activity measured in or over occipital cortex can be modulated by selective attention. Fronto-parietal circuits have been implicated in this modulation, albeit with relatively few causal demonstrations hitherto. But more causal demonstrations of influences from frontal and/or parietal regions upon visual cortex are now forthcoming. These arise from new methodological combinations, including application of neural measures of visual processing to lesioned patients, or in healthy individuals undergoing non-invasive stimulation of particular cortical areas via TMS.

7. Emotional modulation of visual processing

As mentioned earlier, attentional factors related to task-relevance are not the only modulatory influences upon visual processing. The emotional value of a stimulus may also produce some analogous effects (figure 7), typified by relatively enhanced and/or sustained neural responses for emotional relative to neutral stimuli in functional neuroimaging studies (Vuilleumier 2005; Pourtois & Vuilleumier 2006). But the potential sources for imposing these modulations upon visual processing may involve distinct brain structures than attentional control by task-relevance, outside of parietal and frontal cortex. For instance, some converging evidence from both animal and human research suggests that emotion-related modulation of visual processing involves the amygdala, an almond-shaped nucleus in the anterior medial temporal lobe, known to be critically implicated in fear processing and fear-related learning (LeDoux 2000; Phelps & LeDoux 2005), and perhaps other social-affective appraisal processes (Sander et al. 2003; Rosen & Donley 2006).

Figure 7.

Example of emotional modulation of activation in FFA. Results from fMRI in normal subjects show that FFA exhibits selective responses to pictures of faces (green), not to houses (blue), but also exhibits a further increase to emotionally salient facial expression, particularly when fearful or threat-related (red). Data from P. Vuilleumier & G. Pourtois (2005, unpublished); see also Vuilleumier et al. 2001a). Units for fMRI activation (betas) correspond to parameters estimates for event-related changes in BOLD signal.

A longstanding behavioural literature has considered processing of emotion-related stimuli, in both normal and clinical populations (Wells & Matthews 1994; Mogg & Bradley 1998). In the past decade, such work has been supplemented by a growing body of neuroimaging data concerning possible emotional influences upon perceptual processing (e.g. Lang et al. 1990; Lane et al. 1999; Sabatinelli et al. 2005; for review see Vuilleumier et al. 2003). Some of these findings have proved so replicable that they have led to standard procedures in neuroimaging assessments of clinical groups and genetic subpopulations (e.g. Bertolino et al. 2005; Hariri et al. 2005; Meyer-Lindenberg et al. 2005). In particular, fMRI studies of face processing have repeatedly shown that emotional facial expressions can produce significant increases in activation for the amygdala, and also for face-responsive regions within visual cortex (e.g. FFA in lateral fusiform gyrus). Such increases are typically greater for negative facial expressions associated with possible threat, such as fear (Vuilleumier et al. 2001a; Surguladze et al. 2003), although some similar effects have also been reported for other emotional expressions, including positive emotions such as happiness (Winston et al. 2003). Analogously, increased visual activation for complex emotional visual scenes (e.g. mutilations, assaults, etc), relative to neutral scenes, has been found in lateral occipital cortical areas involved in object processing (Lane et al. 1999; Sabatinelli et al. 2005). Such increases in visual activation triggered by emotional information are usually quite specific, affecting those visual regions selectively activated by the current stimulus category, rather than a non-specific arousal effect that influences all brain areas. Thus, in a study where pictures of emotional or neutral faces were presented concurrently with pictures of houses, displays with emotional faces produced an increased activation in face-selective areas (the FFA) but not in PPA house-selective areas (Vuilleumier et al. 2001a).

Analogously, a recent study found (Peelen et al. submitted) that movies showing emotional body movements (relative to neutral body movements) produced selective increases in the extrastriate body area, as well as in the fusiform body area (FBA, that partly overlaps with the FFA but shows body-selective rather than face-selective responses; Peelen & Downing 2005). Within fusiform cortex, increased activation to emotional versus neutral bodies was significantly correlated on a voxel-wise basis with the degree of body-selectivity for that voxel, but was not correlated with face-selectivity (both forms of selectivity being defined relative to a third category of objects). This suggests that emotional signals from seen body movements specifically modulate neural populations involved in processing seen bodies selectively, rather than having a more diffuse arousal-like effect on the visual system. Furthermore, in audition, the emotional prosody of voices has been found to boost activation (relative to neutral prosody) within a restricted region of superior temporal cortex known to be selective for human voices (Grandjean et al. 2005).

Taken together, all these findings show that emotion can produce activations strikingly analogous to those due to selective attention, now enhancing the representation of emotionally relevant (rather than strictly task-relevant) stimuli in specific regions of sensory cortex. One interpretation might be that emotional stimuli are simply more ‘attended’. But we argue below that the findings for emotional stimuli may typically reflect modulation imposed by different circuits than those typically involved in modulations due to selective attention or task-relevance (cf. the frontal and parietal results above).

The data from human neuroimaging showing emotional modulation of visual processing converge with monkey single-cell results, demonstrating for instance that emotional facial expressions (figure 1b) can modulate the response of face-selective neurons in temporal cortex (Sugase et al. 1999). Those single-cell recordings revealed that the initial phase of firing in such neurons was driven by global distinctions such as face–non-face category, while a subsequent phase (starting approx. 50–100 ms later) showed enhanced firing for faces with particular expressions (or with pre-existing familiarity, for some other neurons). Formal information-analyses of neural activity confirmed that the same neurons can carry different information about the same stimulus at distinct latencies, with early activity coding for object category and later activity for emotional value (Oram & Richmond 1999; Sugase et al. 1999). Such emotional boosting of face-selective neurons by expressions and/or affective relevance is reminiscent of the boosting produced by selective attention. But while this has been similarly ascribed to top-down or re-entrant feedback influences from remote brain areas (Sugase et al. 1999; Vuilleumier et al. 2004a), different areas have been hypothesized to play a crucial role for emotional influences, such as limbic regions involved in affect and memory (e.g. the amygdala), instead of parietal or frontal cortex.

Recent EEG recordings in humans also indicate potential analogies between emotion and attention effects on sensory responses. Several studies found a higher amplitude of visual evoked potentials for emotional versus neutral faces (Eimer & Holmes 2002; Pizzagalli et al. 2002; Eger et al. 2003; Ashley et al. 2004), including an enhancement of the P1 component at approximately 120 ms that is thought to be generated in extrastriate cortex (Batty & Taylor 2003; Pourtois et al. 2005), together with later more sustained effects (Krolak-Salmon et al. 2001). Such enhancements of P1 amplitude are often considered as the hallmark for gain modulation of visual processing by selective attention (Hillyard et al. 1998; Martinez et al. 1999). It remains unclear whether this early P1 enhancement for emotional faces may be intrinsically related to processing of emotional facial features in particular (Batty & Taylor 2003), or to other modulatory processes (Moratti et al. 2004; Pourtois et al. 2004). But in either case, the P1 effect demonstrates that some emotional-related modulation may arise in visual cortex, either prior to or concomitant with the processing stages traditionally associated with face perception or object recognition (i.e. approx. 170 ms, when the well-known N170 component specifically related to face-processing arises; Bentin & Deouell 2000; Pizzagalli et al. 2002; Holmes et al. 2003).

8. Specific subcortical sources for emotional influences on visual processing

Which circuits lead to emotion-related modulation of visual processing, as in the examples above? How distinct (or common) are these circuits in relation to those imposing task-related modulations of selective attention? One possible candidate for emotion-related modulations was first highlighted by anatomical tracing studies (Amaral & Price 1984; Amaral et al. 2003), showing dense feedback connections between the amygdala and cortical sensory areas. This led to proposals that such pathways might regulate perceptual analysis of emotional stimuli, particularly when these are threat-related. As indirect support for this idea, electrical stimulation of amygdala nuclei in the rat can produce desynchronization of EEG activity in remote cortical areas, including visual cortex (Kapp et al. 1994; Dringenberg et al. 2001). Furthermore, in rats, lesions of the amygdala after auditory fear-conditioning can suppress a late amplification exhibited by auditory cortex neurons for fear-conditioned tones relative to neutral tones, while leaving the initial auditory response unchanged (Armony et al. 1998).

But perhaps the strongest evidence for a truly causal role of the amygdala in modulation of sensory processing by emotional factors has come from human neuroimaging of patients. A recent fMRI study of patients whose focal lesions could include the amygdala revealed distant functional consequences of these lesions for face processing in visual cortex (Vuilleumier et al. 2004a). In this study, we selected a group of patients with epileptic disease characterized by medial temporal-lobe sclerosis, for whom structural imaging showed that their sclerotic damage involved either the amygdala and hippocampus or just the hippocampus sparing the amygdala. Visual cortex was completely intact in all cases. Both groups of patients performed a task with fearful and neutral faces, in which faces were either task-relevant (attended) or task-irrelevant (unattended), analogous to the attention manipulation shown earlier in figure 2a. Patients with hippocampal damage but intact amygdala showed (figure 8) a normal enhancement in fusiform face-selective areas for fearful versus neutral faces, as found in a healthy control group (figure 6) and in two other previous studies (Vuilleumier et al. 2001a; Bentley et al. 2003). By contrast, patients having the same temporal lobe disease (and medical treatment) but with additional structural damage affecting the amygdala showed no differential responses to fearful versus neutral faces in fusiform cortex (figure 8).

Figure 8.

Effect of amygdala lesion on fMRI BOLD responses to emotional faces in visual cortex (adapted from Vuilleumier et al. 2004a). Patients with medial temporal-lobe sclerosis, whose lesions involved either (a) the hippocampus alone or (b) the hippocampus plus amygdala, performed a picture-matching task similar to that illustrated in figure 2. Seen faces could be fearful or neutral and task-relevant or not. Patients with hippocampal damage alone (a) showed normal activation of fusiform cortex for fearful versus neutral faces (a(ii)), as for healthy subjects also (not shown). Patients with additional damage to the amygdala (b) showed no effect of fearful expression in visual cortex. By contrast, in both patient groups, fusiform cortex was normally activated by attention to task-relevant faces (a(i), b(i)). These results indicate distant functional consequences for visual cortex responses to fearful faces, caused by amygdala damage.

Moreover, there was a significant inverse correlation between the severity of structural amygdala damage and enhancement of fusiform activity by fearful faces, observed selectively within each hemisphere (i.e. greater right amygdala damage led to a decreased boost of right fusiform activity for fearful expressions and analogously for the impact of left amygdala damage on left fusiform activity), without any such correlation between hemispheres (e.g. right amygdala damage did not predict left fusiform activity for fearful faces). This strongly suggests that increases in fusiform activity for fearful faces depend on ipsilateral intra-hemispheric influences from the amygdala upon fusiform cortex. This is consistent with anatomical evidence for such ipsilateral connections (Amaral et al. 2003). A further important feature of our patient fMRI study (Vuilleumier et al. 2004a) was that, in both patient groups, bilateral fusiform face-responsive areas were normally activated by faces relative to houses, and normally modulated by selective attention when the faces were task-relevant (versus task-irrelevant) for the required behavioural discrimination task (cf. figure 2). The latter two effects confirm that visual cortex itself and attentional influences from other regions upon it (presumably arising from fronto-parietal circuits representing task-set control, see above) were still operating normally.

Taken together, these data (Vuilleumier et al. 2004a) provide direct evidence that the amygdala can influence processing in remote visual cortical areas, normally boosting the representation of fear-related faces in fusiform cortex, in a way that is disrupted after amygdala damage. The results also confirm that the emotional influence of fearful expressions in boosting fusiform activation to faces can be separated from (i.e., disrupted by amygdala damage independently from) the attentional effect of task-relevance upon fusiform activations.

More recent work using EEG in similar patients has extended these fMRI data by showing that amygdala damage (but not hippocampal damage) can also reduce or eliminate the enhancement usually found in the P1 component for fearful faces relative to neutral faces (Rotshtein et al. 2006). This finding accords with the fMRI results above, but add temporal specificity that further confirms a role for the amygdala in modulation of a relatively early extrastriate response to fear. Other EEG studies in patients have shown that amygdala damage may also disrupt later cortical modulations by emotional expressions arising approximately 300 ms after stimulus onset (Krolak-Salmon et al. 2001, 2004), while orbitofrontal damage may impair both early and late ERP effects (Ashley et al. 2003).

Future research using intracranial recordings in epileptic patients (with electrodes implanted on clinical grounds, for presurgical assessment) may prove useful for further investigating the exact nature and timing of such distant interactions between areas, in relation to emotional situations (Kawasaki et al. 2001; Krolak-Salmon et al. 2004). Such recordings with implants are usually limited by clinical constraints. Nevertheless, even single-cell recordings can now take place in some patients (Kreiman et al. 2002; Oya et al. 2002), and will shed further light on emotional and attentional modulation of visual processing. In addition, transient electrical stimulation or post-seizure suppression of activity in amygdala or other limbic regions may provide a further tool to identify inter-regional interactions, in patients investigated for epilepsy prior to surgery (Bartolomei et al. 2005; Lanteaume et al. 2006). Non-invasive approaches with TMS may be more limited in this context, due to insufficient access to deep limbic brain structures in the medial temporal lobe and ventral prefrontal regions. Nevertheless, TMS studies of more lateral cortical regions involved in controlling attention in the presence of emotional stimuli (e.g. Dolcos & McCarthy 2006; Dolcos et al. 2006) may provide further opportunities to study consequences of such causal interventions upon neural activations within limbic pathways.

9. Possible relations between attentional and emotional modulation of visual processing

Several behavioural observations have been taken to suggest that emotional stimuli may tend to ‘capture’ attention (Öhman 1986; Mogg et al. 1997; Fox et al. 2000; Vuilleumier & Schwartz 2001a,b; Anderson 2005). Might the emotional modulations of visual processing described here (e.g. increased activation of fusiform cortex for faces with fearful expressions, and so on) perhaps simply reflect enhanced ‘attention’ per se (e.g. Öhman 1986; Ohman et al. 2001; see also Pourtois et al. 2005)? An alternative perspective, that we favour, may be that emotional modulations (e.g. from amygdala) can boost the processing of particular stimuli to give them added ‘competitive strength’ in neural representation (Desimone & Duncan 1995; Kastner & Ungerleider 2001) against other incoming stimuli. This could arise for different reasons (and with different causal neurobiological sources) than attentional signals related to task-relevance per se.

As noted above, fMRI results in temporal-lobe patients (Vuilleumier et al. 2004a) have shown that amygdala lesions can abolish enhancement of the fusiform response for fearful versus neutral faces, yet without disrupting the normal boost (Wojciulik et al. 1998) for task-relevant versus task-irrelevant faces in the same fusiform region for the same patients. Moreover, in normal subjects, we found (Vuilleumier et al. 2001a) that orthogonal manipulations of emotion (fearful versus neutral) or task-relevance (faces versus houses judged) were additive in their impacts on the fusiform response to faces. The amygdala response to fearful expressions did not depend on attention in that particular paradigm. However, some other studies have suggested that, under sufficiently attention-demanding conditions, the amygdala response to fearful faces might be reduced when those faces are unattended (Pessoa et al. 2002): for instance, under conditions of attentional load that are strong enough to eliminate the fusiform response to the faces. But such a pattern might be reconciled with attentional modulations due to task-relevance having different causal modulatory sources than emotional modulations of visual processing (even though both types of modulation can affect the same structure). Such a pattern might simply imply that attentional modulation can sometimes be strong enough (e.g. under high perceptual load, Lavie 2005; Schwartz et al. 2005) to override any apparent emotional influences.

The literature shows that, in several situations, emotional modulations can still arise from task-irrelevant stimuli (e.g. Critchley et al. 2000; Pasley et al. 2004; Williams et al. 2004; Keil et al. 2005). A recent fMRI study (Jiang & He 2006) showed that when face images are rendered invisible by interocular suppression, activity in FFA is reduced to both neutral and fearful faces (although it is still measurable) while activity in the amygdala may be reduced for neutral but not fearful faces. Several EEG studies have manipulated attentional (task-relevance) and emotional (e.g. fearful versus neutral facial expression) factors orthogonally, analogous to our fMRI design described above (Vuilleumier et al. 2001a, 2004a). Holmes et al. (2003) found that task-relevance modulated face processing at a relatively late stage (from approx. 170–180 ms onwards), following the face-specific N170 component (Bentin & Deouell 2000); this was later than the earliest component modulated by fearful expression (P1 component, arising at approx. 120 ms). This appears consistent with some emotional modulation arising in extrastriate visual cortex (the likely source of the P1) prior to task-related attentional selection (Luck 1995), and prior to full completion of cortical face processing. Another EEG study (Keil et al. 2005) used a different paradigm, with emotional scenes presented in either hemifield (right or left) while participants directed attention to detect a target on one or other side selectively. The results again demonstrated additive effects of emotion and attention on visual evoked potentials. Other studies have reported emotional increases in visual ERPs to affective scenes (Sato et al. 2001; Schupp et al. 2003).

Further evidence that may be compatible with separate causal sources for emotional or attentional modulation comes from neuropsychological and neuroimaging studies in patients with right parietal damage and consequent left neglect/extinction. As we described earlier, mechanisms of spatial attention are thought to be pathologically biased in such patients, resulting in perceptual extinction and unawareness of contralesional stimuli when these must compete with ipsilesional inputs (Driver & Vuilleumier 2001; Driver et al. 2004). The fMRI data in such patients show that residual activation evoked by unseen (extinguished) faces in fusiform cortex can still be modulated by the emotional expression of the unseen faces, with a similar boosting of fusiform response to that observed for fearful versus neutral faces when consciously seen (Vuilleumier et al. 2002a). Thus, when a face was presented in the contralesional (left) hemifield of a right-parietal patient, together with a distractor (house) in the ipsilesional right hemifield, fusiform activation was not only found to be greater during conscious detection of the contralesional face (as opposed to trials where it was extinguished from awareness, see earlier sections), but was also greater for fearful than neutral faces regardless of whether the face was seen or extinguished. Fearful expressions also activated the amygdala and orbitofrontal cortex (OFC), again regardless of awareness. Such a pattern echoes the additive effects of emotion and attention that we have observed in healthy subjects (Vuilleumier et al. 2001a); and it accords with the notion that emotional modulation of the fusiform may reflect modulatory signals from amygdala, that can still operate despite deficient spatial attention following damage to parietal systems. Moreover, in keeping with this preserved emotional modulation, parietal patients can show significantly less severe extinction overall for emotional stimuli relative to neutral stimuli (Vuilleumier & Schwartz 2001a,b; Fox 2002), presumably because emotional modulation of visual processing can still enhance the perceptual weight of such stimuli in any competition for awareness, thus partly compensating for reduced attention to contralesional inputs caused by the parietal lesion. Furthermore, unconscious perception of facial emotions has also been found to produce ‘implicit’ or indirect behavioural priming effects in parietal patients with visual extinction, even when they fail consciously to detect the contralesional face (Williams & Mattingley 2004).

In summary, just as attentional modulation of visual processing (due to task-relevance) can have major consequences for perceptual awareness (Mack & Rock 1998; Beck et al. 2001; Driver 2001; Chun & Marois 2002), by providing top-down biases that affect sensory representations of currently task-relevant information, emotional modulations may also analogously affect perception and awareness, by imposing a distinct source of bias upon sensory representations, but now based on signals of affective relevance. Just as frontal and/or parietal damage can disrupt attentional control to dramatically alter perceptual awareness in patients, so too amygdala dysfunction may not only cause difficulties in emotional or affective learning tasks, but may also have some distinct impacts on perceptual processing and awareness (e.g. eliminating detection advantages for emotional stimuli, Anderson & Phelps 2001, see below).

10. Emotional influences on perceptual tasks

Many behavioural studies have shown that, in healthy normal subjects, emotional stimuli can often be detected better and quicker than neutral stimuli. For instance, visual search can be speeded for emotional faces (Fox et al. 2000; Eastwood et al. 2001), while the attentional blink is reduced for emotional words (Anderson & Phelps 2001; Anderson 2005). Spatial cueing effects, in paradigms classically used to study selective attention, can also be enhanced when cues involve emotional or threat-related stimuli rather than neutral stimuli (Pourtois et al. 2004; Phelps et al. in press). But importantly, such effects of emotion on perceptual performance and on associated competitive advantages can be eliminated in patients with amygdala lesions (e.g. Anderson & Phelps 2001), providing further causal evidence that lesions in emotion-related regions of the brain may not only impair emotion processing, but can also have some specific effects on vision, especially in competitive situations with several visual events.

In contrast to amygdala patients, increased rather than decreased perceptual effects of emotional stimuli are often seen in patients with anxiety disorders (Mathews et al. 1990; Mogg et al. 1991; Fox et al. 2001; Yiend & Mathews 2001). Moreover, such individuals may show enhanced amygdala activation to threat-related stimuli (Bishop et al. 2004a,b; Etkin et al. 2004), and increased emotional enhancement in visual cortex (Sabatinelli et al. 2005). Such observations may accord with classic cognitive accounts of anxiety disorders that have emphasized the role of ‘attentional biases’ in these patients (Mathews et al. 1990; Mogg et al. 1991). But here, we emphasize again that the neurobiological sources for emotional modulation of sensory processing may differ from those for attentional modulation in the sense of task-relevance (see above).

Further approaches to such issues in the future will concern pharmacological interventions intended to target potential sources of emotional modulation, separately from potential sources for task-related attentional influences. Initial pharmacological fMRI studies, manipulating cholinergic neuromodulatory transmission, were found to affect attention-related activations but not emotion-related effects in visual cortex (Bentley et al. 2003). Other drugs acting on adrenergic or benzodiazepine systems, as well as neuropeptides (Kirsch et al. 2005) and neurohormones (van Stegeren et al. 2007), might produce important regulatory effects on emotion-related responses instead (Robbins 2007).

Finally, we should note that although we focused for simplicity on just the amygdala in our review of emotional modulation, it is unlikely to be the sole source for emotional modulation of sensory processing (just as the FEF is unlikely to be the sole source for task-related attention, but was likewise emphasized for simplicity above). OFC and the striatum (Robbins 2007; Dolan 2007) are also potential candidates likely to play important roles in emotional or motivational modulation of sensory processing (Ashley et al. 2003). Moreover, there is now some initial evidence that, under some circumstances, reward-related information can modulate parietal neurons thought to play a role in the control of selective attention, or in the direction of saccades (Platt & Glimcher 1999; Maunsell 2004; Sugrue et al. 2004). This could provide a potential functional link for rewards to interface with attentional systems. There is also some new emerging evidence that rewards might directly affect sensory processing in early cortical regions, including even primary visual cortex (Shuler & Bear 2006). For both parietal and sensory cortex, it seems likely that the effective reward signals would probably be conveyed to neurons in parietal or sensory areas by remote brain areas implicated in emotional and motivational processes, such as OFC, striatum and/or amygdala (Cavada & Goldman-Rakic 1989). The sources of such reward-related signals might be identified in future by combining lesion studies with imaging or neurophysiological measures in remote and intact sensory cortices, as we illustrated here for attention and emotion. Moreover, in animals, inactivation or stimulation methods targeting specific nuclei (e.g. in the amygdala, or in distinct regions of OFC) might be performed while recording functional consequences for remote sensory areas.

11. Concluding remarks

We have considered attentional and emotional modulation of visual processing as two complementary examples. In both cases, processing in some regions (e.g. visual cortex) may be causally influenced by remote but interconnected areas (e.g. fronto-parietal circuits for attentional influences; or limbic circuits involving the amygdala for emotional influences). Both kinds of modulation can have analogous consequences at the neural and behavioural levels, including enhanced visual responses and competitive strength for more attended or more emotional stimuli. But while it is often said that emotional stimuli may ‘capture attention’, we suggest that emotional and attentional modulations of vision may often reflect independent effects, imposed by separate circuits, albeit influencing some common regions such as visual cortex. Thus, these findings indicate that sources of top-down biases can provide multiple influences on perceptual systems, rather than there being just a single unified source of modulation (Vuilleumier 2005).

Here, we have illustrated these points by new approaches, including the combination of functional neuroimaging with either lesion studies in brain-damaged patients or online neuro-disruption methods such as TMS, to address causal impacts on remote but interconnected regions. As fMRI can now be applied to studies in monkeys or rodents also, combining such interventional manipulations with neuroimaging may provide an important link between human cognitive neuroscience and animal research in future. These new approaches illustrate the increasing shift in cognitive neuroscience from considering the role of single brain areas in isolation, to a focus on how different regions may influence each other causally. For the particular case of visual processing, such studies reveal that even basic perceptual processing can be substantially influenced by higher-level factors, such as task-relevance or affective status. Although the visual system was once considered to comprise cognitively impenetrable ‘modules’ that would proceed automatically, regardless of higher-level influences, a much more interactive and recursive interplay between different brain systems seems to apply.