The face of fear: Effects of eye gaze and emotion on visual attention

Abstract

We investigated whether a fearful expression enhances the effect of another’s gaze in directing the attention of an observer. Participants viewed photographs of faces whose gaze was directed ahead, to the left or to the right. Target letters then appeared unpredictably to the left or right. As expected, targets in the location indicated by gaze were detected more rapidly. In nonanxious volunteers the effects of fearful gaze did not differ from neutral gaze, but fearful expression had a more powerful influence in a selected high anxious group. Attention is thus more likely to be guided by the direction of fearful than neutral gaze, but only in anxiety-prone individuals.

Observing another’s direction of gaze often results in a congruent focus of attention on the part of the observer. This effect emerges quite early in life, and has been noted in infants as young as 3 months (Hood, Willen, & Driver, 1998). When infants saw faces displayed on a computer whose eyes moved to the left or right, followed by a target stimulus on the same or the opposite side, saccades towards the target were faster if the observed gaze had been in the same direction. This early sensitivity to gaze direction was taken by the authors as consistent with the suggestion that detection of gaze is a biologically prepared ability, possibly controlled by a specialized brain module.

In the experiment to be described here we investigated the influence of eye gaze in guiding attention in adult participants, with particular reference to the influence of facial expression displayed by the model, and the emotionality of the observer. As will be discussed later, we hypothesized that fearful gaze might be especially effective in guiding the attention of an observer to the same location, but particularly so in highly anxious individuals.

We followed the general method described by a number of other researchers, albeit without reference to emotion. In an initial report by Friesen and Kingstone (1998), schematically drawn faces with blanked eyes were displayed centrally for 680 ms, after which the eye pupils appeared, apparently looking straight ahead, to the left or to the right. With a stimulus onset asynchrony (SOA) of 105, 300, 600, or 1005 ms, a target letter (F or T) appeared unpredictably but with equal frequency on the left or right side of the screen. For each SOA except the longest, detection times were significantly faster when targets were in the location consistent with gaze direction, despite instructions that eye gaze was not in fact predictive. There were no significant differences between the conditions when gaze was straight ahead and when targets appeared in the gaze-incongruent location, at any SOA. That is, there were significant benefits of cueing by gaze direction with an SOA of 105-600 ms, but no significant costs when gaze and targets were in opposing locations.

Two further series broadly confirming these findings were reported by Driver, Davis, Ricciardelli, Kidd, Maxwell, and Baron-Cohen (1999) and Langton and Bruce (1999). In the former series, photographed faces were again displayed centrally, with eyes directed to the left or right, followed by one of two letter targets randomly to the left or right, after an SOA of 100, 300, or 700 ms. Participants were instructed to fixate the face, and told that direction of gaze gave no information about where the target would appear. Congruency between eye gaze direction and target location resulted in faster discrimination times, although this was reliable only at 700 ms SOA. In a second experiment the authors changed their procedure so that the face was first displayed with blanked eyes for 900 ms, after which the pupils appeared, thus allowing more time to process the face itself. Results now showed reliable effects at both 300 and 700 ms, but not at 100 ms. Finally, in a third experiment designed to test the extent to which these effects were unintended, the proportion of incongruent to congruent trials was increased fourfold. This change reversed the speeding effect for congruently cued targets at 700 ms, but congruently cued targets were still detected more rapidly at 300 ms.

Rather than varying eye gaze alone, Langton and Bruce (1999) presented photographs of profiles orientated to the left or right, or heads that were tilted up or downwards. With this method, a reliable congruency advantage for detecting a single target was present at 100 ms, but not at 500 or 1000 ms SOA. The effect was little altered by changing the proportion of congruent trials, but was reduced to marginal levels by inverting the photographs. Thus, in both the latter series the overall superiority of target detection speed under conditions of congruency between gaze—or head direction—and target location was confirmed. The explanation for the difference in optimal SOA is not clear, but may be due to a greater speed of processing whole head orientation (and schematic faces), in comparison with eye gaze cues set within realistic photographs of faces. Both research groups concur, however, in suggesting that their data is consistent with a relatively automatic effect, which is difficult to oppose, at least at a short SOA.

These experiments are clearly related to the method introduced by Posner (1978, 1980), in which attention may be directed by the appearance of an object to the left or right of fixation (exogenous cueing) or a central cue, such as an arrow pointing right or left (endogenous cueing). In a typical study, targets appearing in the cued (or valid) location are detected faster than when they appear in the absence of any directional cue (baseline trials). In contrast, when targets appear in a location other than the one that was cued (i.e., the cue was invalid), they are often detected more slowly. These effects are commonly attributed to the benefits of having attention already engaged in the location indicated by valid cues, and/or corresponding time costs of having to disengage from that location and redirect attention to the alternative position. The time course of these effects differs between exogenous and endogenous cues, however, with benefits being typically slower to emerge when cues are endogenous, and costs being less obvious, leading some to suggest that the underlying mechanisms differ (e.g., Cheal & Lyon, 1991).

In all the experiments described earlier, in which eye gaze was used as a central cue, the faces displayed had neutral expressions. However, a number of disparate lines of evidence suggest that faces with certain emotional expressions may guide attention more effectively. For example, in the so-called “face-in-the-crowd” effect, discrepant angry faces are typically found faster than are discrepant happy faces, when each is embedded in a matrix of neutral faces (e.g., Eastwood, Smilek, & Merikle, 2001; Fox, Lester, Russo, Bowles, Pichler, & Dutton, 2000; Hanson & Hanson, 1988; Öhman, Lundqvist, & Esteves, 2001). Hanson and Hanson (1988) claimed that angry faces “popped out” of the crowd, with no increase in search latencies between displays of four to nine faces. More recent studies have found that latencies do increase from small to large displays with angry targets, but critically the search slopes were less than for happy targets (Eastwood et al., 2001; Fox et al., 2000; Öhman et al., 2001). To illustrate, Eastwood et al. (2001) used displays of 6-18 schematic faces, which subjects were required to search in order to find and report on the location of the single example having a discrepant expression. As the number of neutral distracters increased, search time also increased, but these search slopes were quite shallow. More critically, search time overall, and the search time slope, was significantly less for face targets with negative rather than positive expressions. This difference disappeared when faces were inverted. The authors thus concluded that faces with negative expressions capture focal attention more effectively than do positive expressions.

In the present context, this conclusion raises the interesting question of whether similar effects might apply to other forms of attentional guidance, such as the eye gaze effect discussed above. For example, do faces with a fearful expression guide attention more effectively towards their direction of gaze? Although the data just discussed is concerned with attention to the face itself, preferential attention to faces showing negative emotion might also be associated with greater influence of gaze direction. There are some a priori reasons for supposing that a fearful expression may promote attention to gaze direction. In considering possible origins of the general phenomenon of direction of attention by another’s gaze, it seems plausible that such a propensity could convey significant advantages, because the gaze of others often indicates the location of important events. These include motivationally significant objects such as the presence of food and other rewards, or dangers to be avoided. Furthermore, the type of event involved will often be signalled by the other’s facial expression, so that a combination of directed gaze and fearful expression would likely indicate the location of a threat or danger. Selective processing of this type could thus have been favoured through learning or natural selection.

The experiment described here was designed to test the hypothesis that the direction of eye gaze, for both neutral and fearful faces, will influence the location of attention, but that fearful faces may sometimes exert a stronger influence. We initially supposed that fearful expression might always enhance the benefits of gaze congruence in speeding target detection, although without necessarily increasing gaze-incongruent costs (cf. Friesen & Kingstone, 1998). However, in a pilot experiment on 20 members of the Cognition and Brain Sciences volunteer panel, using the method to be described below, we found no difference between the effects due to eye gaze with fearful versus neutral faces. For faces having either type of expression, targets in the location indicated by direction of eye gaze were reliably identified faster than were targets in the alternative location, but the interaction between target location and expression did not approach significance.

Consequently, we considered the possibility that effects of emotional expression might be observed only in those likely to be particularly sensitive to fear-related cues. Although most people are likely to attend cues signalling acute danger, this occurs at lower levels of potential threat in those individuals who are prone to experience anxiety. Thus, people who report high levels of trait anxiety are more likely to have their attention held by negative words, pictures of mildly aversive scenes, or emotional faces (Bradley, Mogg, & Millar, 2000; Mathews & MacLeod, 1994). For example, Bradley et al. (2000) presented pairs of faces, one with a neutral and the other with a threatening (angry) or happy expression, and required subjects to detect a target that replaced one of these faces after an SOA of 500 ms. Regardless of whether or not an eye movement was detected, high trait-anxious individuals were relatively faster than were low anxious controls to detect targets in the prior location of a face with an angry (but not happy) expression.

Two further studies have specifically addressed the contribution of differential engagement of attention with, or disengagement of attention from, emotional cues such as threatening faces or pictures (Fox, Russo, Bowles, & Dutton, 2001; Yiend & Mathews, 2001). Anxiety-related differences were found in both studies (based on divisions by state anxiety in the former and trait anxiety in the latter), and in both cases these differences were confined to conditions in which targets appeared in locations other than those occupied by threatening cues. That is, high trait- or state-anxious individuals appear to take longer than do others to disengage their attention from mildly threatening stimuli.

The foregoing discussion led us to the view that fearful expression alone—in the likely absence of any real danger—may provide too weak a signal to capture attention in low anxious individuals, consistent with previous research (e. g., Bradley et al., 2000; Fox et al., 2001). If so, then the effects of fearful eye gaze—again in the absence of any reason to expect danger—may be seen only in relatively anxiety-prone individuals. Consequently, differential effects of eye gaze due to a fearful facial expression may be more easily detected in high rather than in low trait-anxious groups. If attention in such anxious individuals is more powerfully guided to locations associated with potential threat, they should be faster to detect targets that are congruent with eye gaze direction. Likewise, if the attention of anxious individuals is more powerfully held by locations associated with potential threat, they should be slower to detect targets in locations that are incongruent with fearful rather than neutral gaze.

METHODS

Participants

There were 45 participants, of whom 30 were female, drawn from the undergraduate student population at the University of Essex, and aged between 18 and 24 years. Screening at the beginning of the academic year allowed preferential selection of those with either high or low scores on the trait scale of the State-Trait Anxiety Inventory (STAI; Spielberger, Gorsuch, Lushene, Vagg, & Jacobs, 1983). Eight participants had not been prescreened and so completed the trait scale at the end of the experiment, at the same time that all participants were given the state scale.

The STAI consists of two sets of 20 items, one set of questions about how the respondent feels right now (state anxiety), and the other about how they generally feel (trait anxiety), with each item to be rated on a 1-4 point scale. Minimum score of each scale is thus 20, and the maximum is 80. Norms for US College students (Spielberger et al., 1983) indicate a mean trait score of around 39 (38 for males and 40 for females), with a standard deviation of 9.5 (9 and 10 respectively). The median for the present sample was 40. Preselected participants had trait scores of 45 or more (high group) or 35 or less (low) at screening: At the time of testing three participants had scores lying closer to the median, although only one of these (who scored at the median of 40) was excluded. The final high trait group (N = 22) had a mean trait score of 53 (SD = 7), and the low group (N = 22) had a mean of 31 (SD = 4).

Materials

Photographs of eight individuals, four men and four women, were selected from the Ekman and Friesen (1976) series. Hair and nonfacial areas were masked or removed from the photograph, so that only the central face area was visible, with eyes looking straight ahead. Fearful faces were chosen on the basis that they were readily recognized as representing fear rather than other emotions (Ekman & Friesen, 1976). Pictures of the same individuals were used in the neutral expression trials, so that there were no facial identity differences between conditions other than emotional expression itself (see Figure 1).

Figure 1.

Sample faces used in the experiment, showing eye gaze central with a neutral expression (on the left) and eye gaze directed left with a fearful expression (on the right).

Two new versions of each photograph were then produced, in which the pupils were digitally moved either to the far left or right side of both eyes, and the vacated space refilled, to simulate left or right hand gaze. Photographs were presented in the centre of the screen, subtending a vertical visual angle of 7°.

Targets were the upper case letters T or L, subtending 3° of visual angle, and were centred 5° from the mid point of the screen. The experiment was run on an Apple Macintosh with a 14-inch colour monitor, using PsyScope software (Cohen, MacWhinney, Flatt, & Provost, 1993). Decision latencies and errors were collected using a response box.

Design

The design involved presenting all possible combinations of the eight photographed individuals with neutral and fearful faces, central, left- and right-sided gaze, left- and right-sided targets, and the target letters L or T. Gaze direction was nonpredictive: That is, targets appeared with equally frequency in locations that were congruent or incongruent with gaze direction. Photographs were first presented for 900 ms with eyes looking straight ahead, and then either remained so, or were replaced by the same photograph with eyes shifted left or right, giving the appearance of the person altering their gaze in that direction. The SOA between eyes shifting and appearance of the target letter was either 300 or 700 ms, which, in combination with the factors already listed, made a total of 384 unique trials (8 × 2 × 3 × 2 × 2 × 2). Trials were presented in random order (see Figure 2 for the sequence of events in a trial).

Figure 2.

Schematic representation of the sequence of events presented to participants.

We included the central eye gaze condition both as a baseline condition, and to assess whether fearful faces might in themselves delay response times in the high trait anxious group. However, in retrospect it was apparent that the absence of any eye-movement cue in the central control condition made this data inappropriate as a matched baseline condition, and it was analysed separately from the averted gaze conditions.

Procedure

Participants were given instructions that their task was to discriminate between the target letters T and L as quickly and accurately as possible by pressing the relevant button on the button-box with their thumb and first finger of their dominant hand. These buttons were labelled and arranged vertically to minimize any interference due to the left/right position of targets. Participants were asked to keep their eyes on the central fixation cross that appeared at the beginning of each trial (for 675 ms), and not to respond until they saw a target letter. The experiment began with 10 practice trials, followed by two blocks of testing separated by a rest period.

RESULTS

Central gaze trials

Errors and outlying latencies greater than 1500 ms or less than 100 ms were removed (1.6% of all responses). We then entered latencies from central gaze trials into an ANOVA with within-participant factors of SOA and emotional expression, and a between-participant factor of trait-anxiety group.

There was a main effect of SOA, with faster responses in trials having the longer SOA, 480 ms versus 512 ms, F(1, 42) = 64.17, p < .001. There were no other significant effects, and the main interaction of interest, between anxiety group and emotional expression, did not reach significance, F(1, 42) = 2.59, p < .12. The expected slowing due to fearful versus neutral faces in the high anxiety group, (see Table 1) was thus not confirmed.

TABLE 1.

Means and standard errors for latency, collapsed across SOA, and divided by trait anxiety groups

	Low trait anxiety group			High trait anxiety group
Gaze	Central	Congruent	Incongruent	Central	Congruent	Incongruent
Neutral face	491 (16)	478 (16)	489 (16)	497 (16)	490 (16)	504 (16)
Fearful face	490 (18)	483 (15)	490 (16)	505 (18)	482 (15)	510 (16)

Congruent versus incongruent trials

Mean latencies from gaze congruent and incongruent trials were then entered into an ANOVA with trait anxiety as the grouping factor, and SOA, expression and congruent versus incongruent gaze as within participant factors. There was a significant main effect of SOA as before, F(1, 42) = 198.60, p < .001, but SOA was not involved in any interactions and was thus not considered further (means shown in Table 1 are collapsed across SOA). There was also a main effect of congruency, with faster responses in congruent trials, F(1, 42) = 21.04, p < .001. This was qualified, however, by the only other significant effect, an interaction between congruency, emotional expression, and anxiety group, F(1, 42) = 4.88, p < .051. ¹

Inspection of Figure 3 suggests that this three-way interaction arose, as predicted, because the congruency effect was greater when faces were fearful in the high (but not low) trait group. Separate analysis of responses in trials with fearful expressions revealed an interaction of congruency with anxiety, F(1, 42) = 4.79, p < .05, with larger congruent-incongruent differences in the high than the low trait group, 28 ms versus 7 ms. Analysis of neutral gaze trials revealed only a main effect for congruence, and no interaction with anxiety group, F < 1.

Figure 3.

Latency to discriminate targets (in ms) collapsed across SOA, and divided by trait anxiety groups, for each expression and congruency of gaze.

The above analyses do not reveal the extent to which the difference between congruent and incongruent trials in the high trait-anxious group depends on fearful gaze-congruent speeding, or on incongruent slowing. In the absence of a matched noncued baseline condition, we computed mean speeding on congruent trials due to fearful versus neutral faces (neutral—fearful latency differences), and mean slowing for incongruent trials due to fearful versus neutral faces (fearful—neutral latency differences). We then compared each of these difference scores across trait anxiety groups. For congruent trials, high trait anxious participants showed relatively more speeding due to fearful gaze (8 ms faster) than did the low anxious group (5 ms slower), t(42) = 1.74, p < .05, one-tail. In contrast, incongruent trials with fearful gaze tended to be slower than neutral in both groups (by 6 ms and 1 ms in high and low groups respectively), t(42) = 0.68, n.s.

DISCUSSION

The results reported here confirm earlier findings that the direction of another’s gaze can have powerful effects on the observer’s attention, by guiding it to the same location and thus facilitating the detection of congruent targets. Targets appearing in the location that was signalled by gaze direction were detected faster than were targets in other locations. The present results could thus be attributed to benefits arising from facilitated engagement at the location indicated by another’s gaze. This conclusion would be consistent with other reports suggesting that congruent cues as to target location benefit their detection, sometimes in the absence of equivalent costs for incongruent targets (Friesen & Kingstone, 1998). However, it remains possible that gaze direction could also lead to additional costs due to having to disengage from the location indicated, in order to identify a target elsewhere.

The main novel feature in the present experiments was the use of faces with fearful expressions, in addition to those with neutral expressions. We argued that, because another’s fearful gaze often signals a source of danger, learning and/or natural selection may have particularly favoured attention to the same location. Results with respect to this hypothesis were mixed. In the group of individuals with relatively low levels of anxiety, fearful and neutral gaze had similar cueing effects. However, for reasons discussed earlier, we considered the possibility that the effects of fearful gaze might be greater in a high trait-anxious group. Consistent with this hypothesis, location of attention in highly anxious individuals was more influenced by fearful than by neutral gaze, unlike the findings for the low anxious group. This was true whether the division by anxiety group was based on trait scores alone, or was restricted to those whose state at the time of testing with consistent with this division (see footnote ¹).

It was less clear whether the influence of fearful gaze direction in high anxious groups can be attributed to congruent speeding or incongruent slowing, or both. Only the speeding for congruent fearful versus neutral face trials was significantly different across trait groups (on hypothesis-driven test), consistent with differential engagement at the location of a potential threat. However, the overall interaction involving anxiety groups quite possibly reflects a combination of congruent speeding and incongruent slowing.

Previous work using an exogenous cueing method has revealed anxiety effects that were confined to slowed disengagement of attention from threatening faces or pictures in high anxious individuals, in the absence of any evidence of engagement differences (Fox et al., 2001; Yiend & Mathews, 2001). However, in contrast to this prior work, the present method employed a central cue: Direction of gaze. There is thus no necessary contradiction between the previous findings of anxiety-related differences in disengagement from threat cues, and the suggestion that the present findings may involve anxiety-related engagement differences. Such differential engagement may arise when a signal indicates the location of a potential threat that has not (yet) occurred, while disengagement differences may be more prominent when a threatening stimulus is present in an already attended location.

Intuitively it seems fairly compelling that anxiety is associated with active engagement of attention at the location of possible threat. For example, in a high crime neighbourhood at night it seems quite likely that locations that could conceal potential assailants would be actively attended. Of course, there are many differences between this hypothetical example and the present experiments. However, on the assumption that another’s fearful gaze acts as a directional signal for possible danger, then the finding of differential congruency effects is at least consistent with greater attentional engagement at potential threat locations. By this argument, anxious individuals may be more likely than are others to have their attention guided by signals associated with threat, and to engage locations cued by those signals.

In conclusion, we believe that these data suggest important individual variations in how visual attention is deployed, based on methods that have been taken to provide evidence for the automaticity of neural systems responsive to gaze direction. The emotional nature of a directional cue, and the emotionality of the observer, can apparently modify basic operations of visual attention such as engagement or disengagement. Research in visual cognition could thus usefully take account of relevant emotional and individual difference variables.