Efficient attentional selection predicts distractor devaluation: ERP evidence for a direct link between attention and emotion

Abstract

Links between attention and emotion were investigated by obtaining electrophysiological measures of attentional selectivity together with behavioural measures of affective evaluation. Participants were asked to rate faces that had just been presented as targets or distractors in a visual search task. Distractors were rated as less trustworthy than targets. To study the association between the efficiency of selective attention during visual search and subsequent emotional responses, the N2pc component was quantified as a function of evaluative judgments. Evaluation of distractor faces (but not target faces) covaried with selective attention. On trials where distractors were later judged negatively, the N2pc emerged earlier, demonstrating that attention was strongly biased towards target events, and distractors were effectively inhibited. When previous distractors were judged positively, the N2pc was delayed, indicating unfocused attention to the target and less distractor suppression. Variations in attentional selectivity across trials can predict subsequent emotional responses, strongly suggesting that attention is closely associated with subsequent affective evaluation.

INTRODUCTION

The adaptive control of behaviour in a complex environment relies on mechanisms that enable the efficient selection of task-relevant information, and the simultaneous inhibition of other information irrelevant for a current task set. Selective attention and emotion are both involved in this prioritization of perceptual and response-related processes. Given their shared function in mediating selective processing, it is not surprising that brain imaging studies have uncovered close links between brain mechanisms involved in attention and emotion (e.g., Bush, Luu, & Posner, 2000; Vuilleumier, Armony, Driver, & Dolan, 2001). The emotional salience of stimuli is known to affect sensory processing (e.g., Lang, Bradley, & Cuthbert, 1990; Lang, Bradley, Fitzsimmons, Cuthbert, Scott, Moulder, & Nangia, 1998), the allocation of attention (e.g., Lang, Bradley, & Cuthbert, 1997; Eastwood, Smilek, & Merikle, 2001; Fox, Russo, Bowles, & Dutton, 2001), as well as memory and decision making processes (see Cacioppo & Gardner, 1999, for a review).

Given that there is now conclusive evidence that the emotional significance of visual events can modulate attentional processes, the question arises whether such links between emotion and attention might in fact be bi-directional, with attentional processes also affecting emotional responses. In other words, does directing attention to one stimulus, while simultaneously ignoring another stimulus, have consequences for the subsequent emotional evaluation of these stimuli? Several recent studies have demonstrated that this is indeed the case. For example, Raymond, Fenske, and Tavassoli (2003) asked participants to first report the location of one target stimulus within a simple visual search display consisting of two coloured abstract images, and then to provide an emotional evaluation of one of these images (using a cheerful/dreary dimension). Images previously presented as distractors were not only rated as less cheerful than images previously seen as targets, but also as less cheerful than novel images. In contrast, ratings for previous targets did not differ from ratings for novel items. To explain these findings, Raymond et al. (2003) proposed that during visual search attentional inhibition of irrelevant distractor items is encoded along with the representation of the distractor, and later during evaluation leads to more negative affective judgments.

Further support for the existence of systematic effects of attentional selectivity on emotional evaluation in general, and for the devaluation-by-inhibition hypothesis in particular was provided in additional studies by Raymond and colleagues using multi-item search arrays. Using a temporally segregated ‘preview’ visual search paradigm, Fenske, Raymond, and Kunar (2004) presented a subset of distractors prior to the remaining target and distractor items. Putative top-down inhibition of the previewed distractors (Watson & Humphreys, 1997) was reflected in their devaluation in a subsequent rating task relative to distractors that were not previewed, again suggesting that attentional selection processes have affective consequences. Raymond, Fenske, and Westoby (2005) demonstrated that distractors located near to the target were devalued relative to far distractors, consistent with the view that attended locations are surrounded by a local inhibitory region (e.g., Bahcall & Kowler, 1999). Extending these findings to meaningful social stimuli, Raymond et al. (2005) found that unfamiliar neutral faces that had been presented as distractors in visual search arrays were subsequently rated as less trustworthy than faces previously presented as search targets, with distractors located near to a target again judged more negatively than more distant distractors. Finally, Fenske, Raymond, Kessler, Westoby, and Tipper (2005) demonstrated that faces associated with a no-go cue were judged as less trustworthy than uncued faces, suggesting that response inhibition can also affect the emotional evaluation of stimuli that are spatially and temporally contingent with a no-go signal.

While such behavioural results provide compelling initial support for the hypothesis that selective attentional processing can influence emotional responses, they do not demonstrate unequivocally that the attentional selection of targets and the inhibition of distractors are directly linked to the subsequent emotional devaluation of distractors. To establish the existence of such a direct link, it is necessary to use an independent direct measure of attentional selectivity in order to uncover systematic covariations between attention and evaluative judgments. Distractor devaluation needs to be shown to be associated with variations in attentional processing, with efficient attentional selection linked to negative distractor ratings, and inefficient selection to more positive ratings.

The aim of the present study was to provide such evidence. Event-related brain potentials (ERPs) were recorded in a task similar to the tasks used by Raymond et al. (2003, 2005). On each trial, participants first had to select and respond to one target from a visual search display containing two greyscale faces (each seen with a transparent colour overlay) in the left and right hemifield (visual search task). They then made a trustworthiness judgment for one of these two faces (evaluation task) seen without any colour overlay. Target faces had to be selected on the basis of gender (with male or female faces designated as target in different blocks), while the response was determined by the target’s easily discriminable overlay colour (blue versus yellow; depicted in light and dark grey in Figure 1). Based on the previous findings by Raymond et al. (2003, 2005), it was predicted that faces that were distractors in the preceding visual search display would be judged as less trustworthy relative to target faces.

Figure1.

Example of the sequence of stimuli in each trial. The search display contained one target and one distractor face, which had either a blue or yellow overlay (depicted here in light and dark grey). In mismatch trials, target and distractor faces were differently coloured, whereas in match trials both faces had the same colour. The to-be-rated face was always a greyscale image. All faces were equiluminant.

To obtain an electrophysiological indicator of attentional selectivity during visual search, the N2pc component was measured in response to each bilateral visual search display. The N2pc component is typically elicited at post-stimulus latencies of 200 to 350 ms at posterior electrodes contralateral to the side of a task-relevant visual event, such as a target in a visual search task, and is assumed to reflect the attentional selection of task-relevant events and inhibition of irrelevant distractors (c.f., Eimer, 1996; Luck & Hillyard, 1994; Woodman & Luck, 1999). N2pc amplitudes reflect the difference in ERP activity between electrodes contralateral and ipsilateral to a target, and thus provide a direct measure of the relative distribution of attention in the visual field. The current study focused on the N2pc because this component provides a unique on-line marker of the selective attentional processing of targets versus distractors: Large N2pc amplitudes indicate fully focused attention and effective distractor inhibition, while small and delayed N2pc components are linked to a more diffuse attentional state.

Because target selection in the visual search task was contingent upon a perceptually demanding discrimination between male and female faces, the speed and efficiency of attentional selectivity was expected to vary substantially across trials. On some trials, attentional target selection will be fast and efficient, and distractors will be fully inhibited (reflected by large N2pc amplitudes). On other trials, attention will remain more diffuse and distractor inhibition weak (reflected by small and delayed N2pc components). If selective attentional processing in the visual search task was directly linked to subsequent distractor devaluation, then more negative distractor ratings should be obtained on trials where attentional target selection was efficient (i.e., distractors were successfully suppressed) and large N2pc amplitudes were therefore elicited. In contrast, more positive distractor ratings should be found for trials with diffuse attention and incomplete distractor suppression, as reflected by small and delayed N2pc components.

To apply this logic directly, N2pc components would have to be quantified on single trials in order to establish their relationship with subsequent evaluative judgements. However, due to the low signal-to-noise ratio of ERP components such as the N2pc, measuring this component requires averaging across many trials. In the present study, this problem was circumvented by using the evaluative judgements produced at the end of each trial as a criterion for sorting trials. Separate ERPs in response to visual search arrays were computed as a function of whether targets or distractors faces were subsequently rated as high or low in their trustworthiness. Two predictions were tested. First, if efficient target selection and distractor inhibition were directly linked to subsequent distractor devaluation, a larger and earlier N2pc component should be found in response to visual search arrays for those trials where distractors were later rated as not trustworthy as compared to trials with more positive distractor ratings. Second, if the subsequent emotional evaluation of target stimuli was unaffected by attention, as suggested by previous behavioural findings of Raymond and colleagues, variations in attentional selectivity, as reflected by the N2pc component, should show no such systematic relationship to the ratings of target faces.

METHODS

Participants

Sixteen volunteers (mean age 29.1 years, 5 males) were paid to participate in this experiment. Two of the participants were left-handed, and all had normal or corrected-to-normal vision. The experiment was performed with the approval of the ethics committee of the School of Psychology, Birkbeck College.

Stimuli

1280 face images with neutral expressions and no visible hair were created using the GenHead software (www.genemation.com). Half of the faces were male and the other half female. Faces were converted from RGB colour to 8 bit greyscale using Corel Photo-Paint, and equated for luminance in Matlab (The Mathworks, Natick, MA). Matlab was also used to create one blue and one yellow tinted version of each greyscale face. Each face subtended 3.8° × 4.2° visual angle.

Experimental procedure and design

Stimuli were presented on a computer screen at a distance of 56 cm. Stimulus presentation and behavioural response collection were controlled by E-Prime software (Psychology Software Tools, Pittsburgh, PA). On each trial, participants performed a visual search task followed by an evaluation task (see Figure 1).

In the visual search task, two faces were presented to the left and right of a central fixation cross for 200 ms. One of the faces was male, the other was female, and each could be either blue or yellow. Each face was drawn randomly for each participant and trial from a pool of 1280 faces so that systematic item effects could be eliminated. Participants were instructed to search for the target gender (e.g., male) and report the colour of this target face using the left index and middle fingers to press keys ‘l’ or ‘k’ on a standard keyboard. In half the trials, one face was blue and the other yellow (mismatch trials). In the other half of the trials, both faces were of the same colour (match trials). These two trial types were equiprobable and randomly intermixed. Match trials were included to prevent participants from being able to respond correctly by attending to the distractor, which would have been possible if target and distractor faces had always been of opposite colour. Following the offset of the search display, a blank screen with a central fixation cross was displayed until 1200 ms after the response.

For the subsequent evaluation task, a single face was presented in greyscale for 350 ms at the centre of the screen. This to-be-rated face was equally often the target or the distractor of the preceding visual search display, and appeared 1200 ms after the response in the visual search task. Participants indicated their trustworthiness judgment on a 5-point scale ranging from 1 (‘not at all trustworthy’) to 5 (‘very trustworthy’), which appeared 500 ms after the offset of the to-be-rated face and remained on the screen until a response was recorded. The interval between the rating response and onset of the next visual search display was 1500 ms.

Participants performed ten blocks of 64 trials each, resulting in a total of 640 trials. The designated target face in the visual search task was female in five successive blocks, and male in the other five blocks, with order of target gender counterbalanced across participants. In order to prevent carry-over effects from previous exposure, each face was used in only one trial of the experiment. Participants were instructed to maintain central fixation throughout the experiment and to respond to both tasks as quickly and accurately as possible.

Electrophysiological recording and data processing

EEG was DC-recorded (200 Hz digitisation rate, 40 Hz upper amplifier cutoff frequency) from 23 scalp sites using electrodes mounted in an elastic cap in a modified montage of the International 10-20 system. Electrodes were located at sites Fpz, F7, F3, Fz, F4, F8, FC5, FC6, T7, C3, Cz, C4, T8, CP5, CP6, P7, P3, Pz, P4, P8, PO7, PO8, and Oz. All scalp electrodes were recorded with reference to linked earlobes. A bipolar electrode pair at the outer canthi of both eyes was used to monitor horizontal eye movements. Electrode impedances were kept below 5 kΩ. EEG was epoched into 600 ms segments from 100 ms prior to 500 ms after the onset of the visual search display. Epochs containing blinks, eye movements or movement artefacts were removed. ERPs were averaged for each combination of target position (left versus right) and trial type (mismatch versus match). These averages were then further sorted as a function the subsequent trustworthiness rating on each trial (high: rating 4 and 5; low: rating 1 and 2), separately for trials in which ratings were required for target or distractor faces. Trials where targets or distractors were rated as ‘3’ were not included in these averages.

RESULTS

Behavioural performance

Trials where response times (RTs) exceeded 4000 ms in the visual search task, and 5000 ms in the evaluation task, were excluded (less than 0.5% of all trials), and only trials where target colour was correctly reported were analyzed. Visual search performance was better in match trials than in mismatch trials, with faster RTs (766 ms versus 1052 ms, t(15) = 9.37, p < .001) and lower error rates (1.9% versus 7.4%, t(15) = 5.04, p < .001).

Figure 2 shows mean trustworthiness ratings obtained for faces that had previously figured as targets or distractors in the same trial, separately for mismatch and match trials. In a repeated-measures analysis of variance (ANOVA) for the factors trial type (match versus mismatch) and attention (target versus distractor), a main effect of attention was present, F(1,15) = 7.5, p < .015, as distractors were rated as less trustworthy than targets, thus confirming that attentional target selection affected subsequent emotional responses. There was no trial type × attention interaction (F < 1).

Figure 2.

Mean trustworthiness ratings for target and distractor faces in mismatch and match trials. Error bars represent standard errors of the mean.

N2pc component in the visual search task

To verify the existence of an N2pc in the visual search task, ERPs in response to all visual search arrays were first analysed irrespective of subsequent trustworthiness ratings. Figure 3 displays ERPs obtained at occipital electrodes PO7/PO8 contralateral and ipsilateral to the location of the target face on mismatch (left panel) and match trials (right panel).

Figure 3.

Grand-averaged ERPs elicited by the search display in mismatch (left panel) and match (right panel) trials at posterior electrode sites PO7/8 contralateral (solid lines) and ipsilateral (dashed lines) to the target.

Mean ERP amplitudes at PO7/PO8 were computed within two successive time windows (early and late N2pc: 240-290 and 290-340 ms post-stimulus), and were analysed in repeated-measures ANOVAs with the factors trial type (match versus mismatch) and contralaterality (contralateral versus ipsilateral hemisphere relative to the side of the target). Main effects of contralaterality were present for both time windows (F(1,15) = 6.8 and 12.7, p < .02 and .003, respectively), thus confirming that an N2pc was reliably triggered during the attentional selection of target faces. Although the N2pc appeared more pronounced in mismatch trials, no trial type × contralaterality interactions were present in either time window (both F < 1.5).

Figure 4 shows ERPs to visual search displays on trials where subsequent ratings were later required for the distractor (left) or the target (right), pooled across match and mismatch trials, as a function of trustworthiness judgment (low: 1 or 2 on rating scale, bottom panels; high: 4 or 5 on rating scale, top panels).

Figure 4.

Grand-averaged ERPs elicited by the search display at posterior electrode sites PO7/8 contra- and ipsilateral to the target, for trials where the distractor face (left side) or the target face (right side) was subsequently rated, shown as a function of rating (high: 4 or 5, top panels; low: 1 or 2, bottom panels). Waveforms are pooled across mismatch and match trials.

Differences in the judged trustworthiness of the distractor were mirrored by systematic N2pc differences, with an earlier N2pc for trials where distractors were later judged as not trustworthy. In stark contrast, subsequent target ratings showed no apparent relationship to the N2pc in response to visual search displays.

This was confirmed in an analysis of N2pc peak latencies, which were computed by subtracting ERPs at PO7/PO8 ipsilateral to the target from contralateral ERPs, and then determining the latency of the maximal negative peak between 240 and 340 ms post-stimulus for each participant, and separately for trials with high or low target or distractor ratings. While no difference in N2pc peak latencies was observed between trials with negative versus positive target ratings (299 and 301 ms post-stimulus, respectively; F < 1), a significant latency difference was present as a function of distractor ratings, as the N2pc peaked earlier for trials with subsequent negative distractor evaluations relative to trials with positive distractor ratings (285 versus 307 ms; F(1,15) = 6.37, p < .023).

This pattern was further confirmed in repeated-measures ANOVAs conducted for ERP mean amplitudes, separately for trials with distractor or target ratings, for the factors rating (high versus low) and contralaterality. When distractors were rated, a significant rating × contralaterality interaction was present for the early N2pc window (240-290 ms post-stimulus; F(1,15) = 5.4, p < .035). Follow-up analyses revealed a highly significant effect of contralaterality on trials with subsequently low-rated distractors, F(1,15) = 9.3, p < .008, reflecting the presence of a robust early N2pc, that was entirely absent for trials where distractors received high ratings (F < 1). In the 290-340 ms time window, a main effect of contralaterality, F(1,15) = 6.5, p < .022, was found without any rating × contralaterality interaction (F < 1), as the later part of the N2pc was present regardless of how distractor faces were subsequently rated (Figure 4). For trials including target ratings, main effects of contralaterality were found for both time windows, F(1,15) = 5.3 and 6.8, p < .036 and .019, respectively, without any indication of rating × contralaterality interactions (both F < 1), suggesting that the efficiency of attentional target selection, as reflected by the N2pc, had no impact on the subsequent rating of target faces.

DISCUSSION

The aim of the present study was to investigate the hypothesis that links between emotion and attention are bi-directional. While it is well known that the emotional salience of stimuli affects attentional processes (e.g., Lang et al., 1997; Fox et al., 2001), the question whether attentional selectivity also affects emotional responses has only recently begun to be addressed (e.g., Raymond et al., 2003). If attention modulates emotion, the attentional selection of targets among distractors in a visual search task should directly affect the subsequent emotional evaluation of these stimuli. Using the N2pc as an electrophysiological index of the relative distribution of attention across target and distractor faces in a bilateral visual search array, it was demonstrated that variations in the efficiency of attentional target selection predict subsequent trustworthiness ratings for distractor faces, but were entirely unrelated to the subsequent rating of target faces.

Similarly to Raymond et al. (2005), participants had to judge the trustworthiness of faces they had previously attended or ignored. Distractor faces were rated as less trustworthy relative to target faces, thereby confirming results from previous experiments. The gender-based attentional selection of target versus distractor faces in the visual search task was reflected by an N2pc component that was present not only on mismatch trials (where the two faces in the visual search array differed in colour, and targets thus needed to be identified to select the correct response), but also on match trials (where both faces had the same colour, and response selection could have taken place without any attentional selection and identification of target faces), suggesting that attention was directed to the face targets in both types of trials.

Most importantly, when ERPs in response to visual search arrays were sorted as a function of subsequent evaluative judgments, a clear dissociation in the affective consequences of target selection and distractor inhibition was obtained. Differences in the judged trustworthiness of previous target faces were not linked to any differences in attentional selectivity across trials, as reflected by the N2pc triggered during the visual search task. In other words, trial-by-trial variations in the efficiency of attentional target processing did not have any effect on subsequent evaluative judgments of target faces. This corresponds perfectly to the finding of Raymond et al. (2003) that affective judgments in response to previously attended items did not differ from judgments to novel items. In marked contrast, when ERPs were sorted as a function of trustworthiness ratings for distractor faces, a systematic relationship between attentional selectivity and subsequent evaluative judgments was revealed. The N2pc emerged earlier on trials where distractor faces were subsequently judged to be untrustworthy relative to trials where trustworthiness ratings were high. In other words, distractor faces were evaluated negatively on trials where attention showed a strong early bias towards target events (implying more effective distractor inhibition), but more positively on trials where the distribution of attention was initially more diffuse, and distractor suppression therefore less pronounced. This pattern of results confirms previous observations by Raymond and colleagues that the attentional selection of targets in visual search tasks is linked to distractor devaluation. Most importantly, it demonstrates the existence of a systematic covariation between an electrophysiological index of attentional selectivity and subsequent evaluative judgments.

It might be argued that variations in trustworthiness judgments produced at the end of each trial may have been due to systematic differences in the physical characteristics of individual faces. Because faces that are low in trustworthiness are known to trigger increased amygdala activations (Winston, Strange, O’Doherty, & Dolan, 2002), faces rated as untrustworthy may thus have been generally more emotionally and attentionally salient during the attentional search task. However, the fact that the assignment of individual faces as targets or distractors was performed in a random fashion for each participant ensured that faces seen as distractors by some participants were seen as targets by others, thereby effectively ruling out the possibility of any such systematic item effects. In addition, the fact that the N2pc emerged earlier on trials where distractors were later judged as untrustworthy cannot easily be explained by assuming that these distractors were more salient, as this should have attenuated rather than enhanced any early attentional bias towards the target. Finally, if residual item effects were responsible for the pattern of ERP results found in the present study, this should have equally applied to trials where target faces had to be rated. However, no effect of trustworthiness ratings on the N2pc was found for these trials.

In summary, the present study has provided new electrophysiological evidence for the existence of systematic covariations between electrophysiological measures of attentional selectivity and behavioural measures of subsequent affective evaluation processes. The efficiency of attentional selectivity in visual search covaries with, and therefore can predict, subsequent emotional responses to distractor stimuli. This pattern of results provides new evidence for the hypothesis that links between attention and emotion are bi-directional, as it suggests that selective attentional processing has immediate consequences for the affective evaluation of visual stimuli.