Abstract
We report a novel effect in which the visual perception of eye-gaze and arrow cues change the way we perceive sound. In our experiments, subjects first saw an arrow or gazing face, and then heard a brief sound originating from one of six locations. Perceived sound origins were shifted in the direction indicated by the arrows or eye-gaze. This perceptual shift was equivalent for both arrows and gazing faces and was unaffected by facial expression, consistent with a generic, supramodal attentional influence by exogenous cues.
Keywords: perceptual contagion, mimicry, attention, sound localization, multisensory
1. Introduction
Our brains are pervasively multisensory. Nearly every brain area examined thus far seems to be driven and/or modulated by multiple sensory modalities [1,2]. Using prior knowledge, these modalities can be integrated to determine the most probable environmental source of a sensory event, and thus mediate adaptive action [3]. For example, detection of a sensory transient is a strong indicator that information about a region of space is becoming outdated, and these signals often evoke a powerful overt or covert orienting response [4] which, in turn, influences our sensory perceptions [5,6]. These exogenous spatial orienting responses transcend individual senses and appear largely supramodal [7].
Humans, and indeed many other creatures, quickly and reflexively attend in the direction cued by (i) an observed individuals' orienting behaviour [8], (ii) conventionalized iconic cues (e.g. arrows [9,10]), and (iii) other salient events [4]. The crossmodal efficacy of exogenous attention, as evoked by auditory and visual transients, has been controversial in part because many studies focused only on reaction times, finding that detection latencies sometimes appeared insensitive to the distribution of spatial attention [7]. For example, several studies have reported a consistent asymmetry in audiovisual attentional cuing in which auditory cues enhanced visual detection, but not vice versa. We explored whether observed visual eye-gaze or arrow cues exert crossmodal influences not only by decreasing reaction times to the detection of spatially congruent sounds, but also by changing perception, as operationalized through categorical report.
In the present study, we tested whether arrows [9,10] and gazing faces [11,12]—both visual cues which reflexively orient attention—influence perceptual judgments regarding the location of a sound. In our experiments, human participants were asked to report the location of a broadband noise stimulus immediately after viewing arrows or gazing faces with neutral, fearful or angry expressions. Under these conditions, subjects might exhibit no influence of attention on sound localization judgments, or might experience (i) elevated error rates favouring the cued direction, (ii) reaction time savings for validly cued targets, and/or (iii) an actual perceptual shift in the category boundary dividing left and right sound locations. Each of these effects was evident and was similar across conditions. Observed visual attentional cues change the way we perceive sound including dissociable effects on reaction time and spatial categorization. Moreover, eye-gaze and arrow stimuli are equally effective, consistent with common origination in learned associations across developmental experience.
2. Experiment 1: observed arrows
(a). Method
(i). Participants
The 11 subjects (five females; age 22 ± 3.2 years) were recruited from the Princeton University community and paid $12 for their participation (an additional subject was excluded prior to analysis because he misunderstood the instructions). All were right-handed, had normal hearing and had normal or corrected-to-normal vision.
(ii). Visual stimuli
Double-headed arrows were generated in Adobe Illustrator and Photoshop (Adobe Systems, Inc), and were designed to be centrally presented, luminance-balanced, and intermediate in contrast and length. Arrows subtended the central 6° of vision. The size of the arrowheads was matched in size to the sclera of the eyes, and the length was matched to the average inter-ocular distance, of the face stimuli used in experiment 2 mentioned below. The contrast between the black background and arrow colour was identical to the contrast between the black background and the whites of the eyes for the face stimuli. A high-contrast white fixation-cross served as a neutral cue.
(iii). Sound stimuli
A noise burst, 30 ms in duration, was chosen for its lack of social relevance and its ease of localizing. Six sound locations along the azimuthal plane were simulated by manipulating inter-aural level cues (or ‘pan value’) [13,14]. A pan value of 0 was equally loud in both ears, while a pan of −1 was heard from the immediate left, and +1 from the immediate right. Pan values of ±0.1, ±0.075 and ±0.025 were used in this study, ranging from relatively easy to moderately difficult to localize as coming from either the left or right. Acoustical stimuli were played to subjects through AKG Acoustics K-240 Semi-Open Studio Headphones. Headphones were used to eliminate the effects of room echo and precisely control inter-aural sound properties [14].
(iv). Design and procedure
We implemented the pan values in a variant of the Posner cuing paradigm [4] using presentation (Neurobehavioral Systems, CA) stimulus control software on a standard PC. The sound stimuli occurred 300 ms after the onset of a visual cue. The visual cue consisted of either a neutral cross or a double-headed arrow pointing to the left or right (figure 1a). Randomized trials consisted of the following conditions: congruent, in which the arrows pointed in the direction of the sound source, incongruent, in which they pointed away, or neutral, displaying a fixation cross. Subjects were instructed to gaze steadily towards the screen and, as quickly and accurately as possible, indicate by button press whether the sound came from their left or right. At this time, the arrow disappeared. Two minute breaks were inserted after 151 and 351 trials to reduce participant fatigue. Total duration was approximately 35 min.
Figure 1.
Experiment 1: arrows. (a) The order and pairing of the trials were randomized for each participant. During each trial, a visual stimulus appeared on a black screen for 300 ms, at which point a sound played from one of three leftward or three rightward pan values. Participants completed the trial by indicating whether the sound came from their right or left; cues disappeared at this time. (b) Observed portion of trials in which subjects localized the sound as coming from the right, as a function of the actual sound pan value. Subjects indicated hearing a sound to their right more often when they saw arrows pointing towards the right, even if the sound stimulus was from the left. The opposite pattern was seen when viewing leftward pointing arrows. Both horizontal (perceptual) and vertical (biasing) shifts were evident (red line, arrow right; black line, neutral; blue line, arrow left). (c) Changes in both perceptual shift (Δα) and response bias (Δb) were significant. No significant changes occurred in task performance, as measured through the β and scale (S) parameters. (d) Repeated measures ANOVA demonstrated a significant main effect of sound eccentricity and a trend towards an effect of congruence on reaction time, with no significant interaction. Post hoc testing showed that congruent trials (green line) were significantly faster than neutral or incongruent trials (yellow line), and that neutral trials (black line) were significantly faster than incongruent trials.
(v). Analysis
We estimated the psychometric function for each subject and cue based on the probability of ‘rightward’ responses at each stimulus pan value. Each psychometric function was a sigmoidal curve, modelled as a cumulative Gaussian in Matlab (The MathWorks Inc., CA) using the maximum-likelihood method (psignifit, http://www.bootstrap-software.org/psignifit/) [15]. Such functions have four free parameters—α, β, γ and λ—which relate, respectively, to the perceptual category boundary, the boundary sharpness, and the minimal and maximal rates with which a response is evoked. One can imagine several ways in which subjects' behaviour might change in response to the arrow cues. The cue might bias their perception of the sound, changing the category boundary, or it may trigger responses based on cue direction rather than sound direction, triggering a stimulus-independent change in rightward response probability. Additionally, the cue might attract attention to the task or distract attention from hearing to vision, increasing or decreasing the error rate without inducing any systematic, directional shift in subjects' responses. Such accounts can be discriminated using the four fit parameters. Changes in α result in a horizontal shift in the sigmoid, indicating a change in perceptual categorization: a decrease in α for right versus left arrow cues corresponds to a perceptual shift of sound location in the direction of the arrow. By contrast, stimulus-independent motor bias (b) in the direction of the arrow can be calculated by comparing γ and λ, which represent the levels of perceptual responses at the extremes of the sigmoid, using the equation b = (1 + γ − λ)/2. If the arrow cue draws attention to the task or distracts from the auditory stimulus, then boundary sharpness and random error rates may change. Sigmoid slope (boundary sharpness) is approximately inversely proportional to β, the standard deviation of the cumulative Gaussian from which the sigmoid is derived. Sigmoid scale (resistance to random error), meanwhile, decreases with γ and λ per equation S = (1 − γ − λ). These parameters thus distinguish multisensory perceptual changes from simple response biases or changes in task performance. To ensure subjects responded to the relevant cue, trials with reaction times faster than 200 ms after sound onset were excluded from analysis as anticipations (5.14% of trials).
(b). Results and discussion
Our results indicate that observing arrow cues changed sound perception by shifting both perceived and reported sound origin in the cued direction (figure 1b). Psychometric functions shifted horizontally, consistent with a change in category boundary (Δα, left-cued versus right-cued, t10 = 3.04, p = 0.012, 95% CI (0.006, 0.04)), and vertically, suggesting that perceptual changes were accompanied by a stimulus-independent response bias (Δb, right-cued versus left-cued: t(10) = 3.20, p = 0.0094, 95% CI (0.01, 0.06)); Δβ and ΔS, gaze-cued versus neutral, n.s., t10 = −0.95, p = 0.36, 95% CI (−0.02, 0.009) and t10 = −2.17, p = 0.055, 95% CI (−0.07, 0.0009)) (figure 1c). Overall, error rates were 12.2 per cent for neutrally cued trials and 13.9 per cent for directionally cued trials.
We analysed reaction times for correct trials using a 3 × 2 repeated-measures ANOVA modelling the effects of sound eccentricity (absolute pan value) and stimulus–response congruence (figure 1d). There was a significant main effect of eccentricity (F2,98 = 10.52, p = 7.9 ×10−5) but only a trend towards an effect of congruence (F2,98 = 2.32, p = 0.10) and no significant interaction (F4,98 = 0.04, p = 1). Post hoc tests revealed that reaction times in congruently cued trials were faster than both incongruent trials (by 53 ms, t10 = 3.51, p = 0.0056, 95% CI (19.39, 87.78)) and neutral trials (by 28 ms, t10 = 3.52, p = 0.0056, 95% CI (10.10, 44.95)); neutral trials were faster than incongruent trials (by 26 ms, t10 = 2.74, p = 0.021, 95% CI (4.80, 46.32)).
These data replicate several prior studies that demonstrated the ability of non-predictive iconic cues to influence orienting reaction times [9,10]. Importantly, however, we found that this influence comprised two distinct effects, one on perceptual categorization and one on reaction time (ANOVA effect by assessment × eccentricity, F2,65 = 3.99, p = 0.024). First, arrows significantly shifted subjects' perception of sound origin in the cued direction; the categorization of sounds closest to the boundary were the most likely to change. Second, reaction times were faster for congruently cued trials than neutral and for neutral than incongruently cued trials. Cue-induced changes in categorization depended on proximity to the category boundary (ANOVA categorization gain by eccentricity, F2,32 = 5.56, p < 0.0088), but changes in reaction time depended only on stimulus–response compatibility (ANOVA reaction time savings by eccentricity, F2,32 = 0.37, p = 0.6951; figure 3a).
Figure 3.
Dissociation between categorization and reaction time. Cue effects on categorization (grey), but not reaction time (black), were influenced by the eccentricity of sound stimuli. This pattern was consistent between (a) arrows and (b) averted gaze cues. These data indicate that attentional cues differentially influence categorical report and reaction time: specifically, categorical report was most strongly influenced by irrelevant crossmodal cues when the categorization is difficult; by contrast, reaction times simply reflected stimulus–response compatibility and were uninfluenced by task difficulty. (a,b) Black line, reaction time; grey line, perceptual report.
Because symbolic communication is a recent human development, we next explored whether similar perceptual effects were exerted by observed eye gaze cues, which exert potent [11,12], supramodal [16] and evolutionarily conserved [17] effects on attention.
3. Experiment 2: observed eye gaze
(a). Method
(i). Participants
Twelve subjects (six females; age 20.2 ± 2.9 years) were recruited from the Princeton University community and paid $8 for their participation. All were right-handed, had normal hearing and had normal or corrected-to-normal vision.
(ii). Visual stimuli
A protractor, laser pointer and digital camera were used to gather colour photographs of two volunteers (males, ages 31 and 36) with expressions neutral, faces towards the camera, and eyes averted 30° left or right (figure 2a). Photographs were edited in Photoshop CS2 (Adobe Systems Inc., CA) to isolate the face on a black background. A white fixation cross, also created in Photoshop, served as a directionally neutral stimulus. Face stimuli were approximately 11° tall (500 pixels of 1280 × 1020 pixel LCD display, 34 cm by 27 cm, at a distance of approximately 65 cm). The same white fixation-cross used in experiment 1 served as the directionally neutral stimulus.
Figure 2.
Experiment 2: eye gaze. (a) Experimental design was identical to experiment 1, except in that gazing faces were shown instead of arrows. Faces were scaled so that the eye region was the same size as the arrows of experiment 1. (b) Gaze-induced psychometric shifts were similar to those evoked in the first experiment, including a significant horizontal and trending vertical shift (red line, gaze right; black line, neutral; blue line, gaze left). (c) Perceptual shifts were significant (Δα), while response bias (Δb) was evident at the trend level. No nonspecific changes were evident in task performance (Δβ and ΔS). (d) Repeated measures ANOVA demonstrated significant main effects of sound eccentricity and congruence on reaction time, but no significant interaction. Post hoc testing showed that incongruent pairings were significantly slower than congruent pairings (green line), and that neutral pairings (black line) were significantly slower than congruent pairings but not significantly faster than incongruent pairings (yellow line).
(iii). Design and procedure
Each subject saw only one of the two model faces. The experimental procedure was identical to experiment 1. Each pairing of three visual and six auditory stimuli was repeated 28 times for a total of 504 trials. Sessions were approximately 30 min in duration. Trials of 5.49 per cent were excluded from analysis as anticipations.
(b). Results and discussion
Observed gaze strongly shifted sound localization perception (figure 2b). Psychometric functions shifted horizontally, consistent with a change in category boundary (Δα, left-cued versus right-cued, t11 = 4.31, p = 0.0012, 95% CI (0.02, 0.05)). There was also a vertical shift, suggesting that perceptual changes were accompanied by an additional, stimulus-independent response bias (Δb, right-cued versus left-cued, trend only: t11 = 2.05, p = 0.0647, 95% CI (−0.001, 0.04); Δβ and ΔS, gaze-cued versus neutral, n.s., t11 = 0.53, p = 0.6, 95% CI (−0.01, 0.01) and t11 = −1.42, p = 0.19, 95% CI (−0.05, 0.01)) (figure 2c). Overall, error rates were 14.4 per cent for neutrally cued trials and 15.1 per cent for directionally cued trials. These results indicate that observing a gaze cue changed sound perception by shifting the perceived sound origin in the direction of gaze.
We analysed reaction times for correct trials using a 3 × 2 repeated-measures ANOVA modelling the effects of sound eccentricity (absolute pan value) and stimulus–response congruence (figure 2d). There was a significant main effect of eccentricity (F2,99 = 28.89, p = 1.3 × 10−10) and of congruence (F2,99 = 6.82, p = 0.0017) without significant interaction (F4,99=0.13, p = 0.97). Post hoc tests revealed that reaction times in congruent trials were faster than both incongruent trials (69 ms, t11 = 4.31, p = 0.0012, 95% CI (33.66, 104.05)) and neutral trials (46 ms, t11 = 5.5824, p = 0.0002, 95% CI (27.92, 64.28)). Neutral trials were faster than incongruent trials only at the trend level (23 ms, t11 = 1.94, p = 0.079, 95% CI (−3.12, 48.64)).
Thus, observed gaze significantly influenced subjects' spatial hearing, shifting perceptions of sound origin in the direction of gaze. Sounds closest to the category boundary were most influenced by gaze (ANOVA categorization gain by eccentricity, F2,35 = 18.18, p < 10−5). Although the data suggest that subjects also exhibited a sound-independent increase in cued responses, this trend did not reach statistical significance. No other changes in the psychometric response function were significant. Reaction times were faster for congruently cued than neutral or incongruently cued trials. However, unlike the perceptual shifts (ANOVA effect by assessment × eccentricity, F2,71 = 10.85, p = 0.0001), reaction time differences did not interact with stimulus eccentricity (ANOVA reaction time savings by eccentricity, F2,35 = 0.72, p = 0.50; figure 3b).
There were no significant differences between participant responses in experiment 1 (arrows) versus experiment 2 (gaze) (p > 0.2 for α, β, b, S and reaction times). Observed cues, whether arrows or averted eye gaze, induced both perceptual shifts and stimulus–response compatibility effects. It is important to note that in our natural environments, both types of cue reliably predict points of interest: the former explicitly, owing to strategic communication, the latter implicitly, owing to correlated environmental responses by related individuals [18]. Modern humans regularly encounter these cues, and this extensive developmental experience may produce a supramodal effect of cuing on perception that operates similarly to other multisensory interactions: that is, by producing Bayesian constraints on the probably provenance of incoming sense data [3].
It has been reported that humans both physically (motor contagion, reviewed [19]) and mentally (emotional contagion, reviewed [20]) entrain to observed individuals. Because gaze and facial expressions are processed reflexively [8,21] and integratively [22,23], we performed a third experiment to determine whether facial affect could enhance cued influences on perception.
4. Experiment 3: emotionally expressive gaze
(a). Method
(i). Participants
Fourteen subjects (nine females; age 19.6 ± 0.8 years) were recruited from the Princeton University community and paid $12 for their participation; 13 were right-handed, and all had normal hearing and normal or corrected-to-normal vision.
(ii). Face stimuli
The same gaze models were photographed as per experiment 2, this time with neutral, angry or fearful expressions (figure 4a). Facial expressivity was validated in a sample of 21 respondents: faces were sequentially presented for categorization as neutral, angry, fearful, disgusted, happy, sad or surprised. Face stimuli used in this experiment were correctly categorized as ‘neutral’, ‘angry’ or ‘fearful’ by at least 70 per cent respondents.
Figure 4.
Experiment 3: emotionally expressive gaze. (a) Left- and right-gazing faces with neutral (left column), angry (middle column) or fearful affect (right column) were acquired from two volunteers. Each participant saw faces from one of the two cues, interleaved in random order across trials. (b) Observed psychometric responses to each cue pair. Gaze-induced psychometric shifts were similar to those evoked in the prior experiments, and were evident for each pair of unique gaze cues (grey line, gaze right; black line, gaze left). (c) Gaze-induced perceptual shifts and (d) gaze-induced response bias, by facial expression. No significant differences were observed between facial expressions. Perceptual shifts and motor biases were significant for all emotional conditions. Asterisks in (c,d) indicate significant difference from zero (p < 0.05).
(iii). Design and procedure
Procedure was identical to experiment 2 except that now each combination of the six auditory stimuli and six visual stimuli was repeated 25 times in random order, for a total of 900 trials over 45 min. To reduce participant fatigue, 2 min breaks were inserted after 200, 400, and 800 trials. Total duration was approximately 45 min. Trials of 6.41 per cent were excluded as anticipations.
(b). Results and discussion
Observed gaze direction strongly influenced auditory localization. Perceptual shifts were similar to those evoked in the previous experiments and were again accompanied by a stimulus-independent response bias (figure 4b). Both effects were significant (Δα, left-cued versus right-cued: anger, t13 = 3.43, p = 0.0045, 95% CI (0.01, 0.03); fear, t13 = 2.45, p = 0.03, 95% CI (0.003, 0.04); neutral, t13 = 2.24, p = 0.043, 95% CI (0.001, 0.03). Δb, right-cued versus left-cued: anger, t13 = 2.32, p = 0.038, 95% CI (0.003, 0.08); fear, t13 = 2.83, p = 0.013, 95% CI (0.009, 0.07); neutral, t13 = 3.32, p = 0.0056, 95% CI (0.02, 0.09); figure 4c). Psychometric shifts induced by observed gaze were unaffected by facial expression (Δα, left-cued versus right-cued, by emotion: ANOVA p = 0.78; Δb, right-cued versus left-cued, by emotion, ANOVA p = 0.70; β, averaged by emotion, ANOVA p = 0.77; S, averaged by emotion, ANOVA p = 0.96).
Overall, error rates were 17.8 per cent for affectively neutral trials, 16.8 per cent for angry cue trials, and 17.3 per cent for fear cue trials. Reaction times were analysed for correct trials using a 3 × 2 × 3 repeated-measures ANOVA modelling the effects of sound eccentricity, congruence, and facial expression. The ANOVA revealed a significant main effect of eccentricity (F2,234 = 9.97, p = 0.0001) and congruence (F1,234 = 8.87, p = 0.003), as in experiment 1, but no effect of facial expression (F2,234 = 0.02, p = 0.98), nor any interaction between factors (all F4,234 = 0.02, p > 0.9) on reaction times. Congruent trials were, on average, 41 ms faster than incongruent (t13 = 3.43, p = 0.0045, 95% CI (15.11, 66.71)).
We replicated the main effect of observed gaze direction on sound localization, but found no effect of facial expression. Perceptions of gaze and expression are known to interact [22,23], however, gaze cuing effects appear insensitive to accompanying facial expression (e.g. [24]). While some studies have shown an influence of emotion on gaze following, they typically found these effects under constrained circumstances or when looking at specific populations having heightened sensitivity to specific emotions (e.g. [25]). An emerging account suggests that the fastest cuing effects are relatively immune to contextual modulation [8,26], suggesting that the perceptual shifts reported here are more strongly tied to these fast effects rather than to slower, more cognitive components which integrate facial expressions.
5. General discussion
Brains evolved to guide behaviour in an interactive world monitored through myriad sensory pathways. Traditionally, primate brains have been understood to be personal and private, and segmented into modules that process distinct sensory modalities [27], that make decisions by associating information between modalities and motivational systems, or that respond to events by coordinating bodily actions [28]. Recent lines of evidence, however, suggest our brains are also deeply integrative, blending information gained through different sensory pathways [2]. In three experiments, we showed that subjects uniformly shift their report of a sound's location in the direction of an arrow or gazing face. We now consider possible mechanisms and neural pathways that may mediate this effect.
Perceptual shifts appear to be a general consequence of attention (cf. [6] for similar results using exogenous cues). In contrast to the consistent asymmetry in audiovisual attentional cuing and enhancement of detection found in previous studies [29,30], we find that both seen arrows and gaze decreased reaction times to spatially congruent sounds. These reaction time savings are invariant with respect to the strength of unimodal evidence. Conversely, we find that cue-induced shifts in categorical report are considerably sensitive to the strength of unimodal evidence, consistent with a perceived shift in sound origin. These findings suggest that observed eye-gaze and arrow cues exert fast, reflexive and supramodal effects on elementary perception. Such cues probably exert their influence by increasing the gain on sensory processing associated with attended regions of space [5,6].
While we believe that these effects reflect a change in internal state, it is possible that perceptual shifts may be triggered by postural changes associated with overt orienting. Overt and covert orienting systems are tightly linked, and attentional shifts are often associated with orienting movements: for example, perception of gaze cues can cause overt mimicry or, with sustained fixation, microsaccadic drift in the direction of attention [17]. These subtle, inducible changes in the subjects' gaze direction can influence their processing of acoustical events [31,32], with neural activity in both visual and auditory areas of the temporal lobe influenced by a subject's eye position [33,34]. Moreover, in a sound localization task manipulating subjects' gaze direction, Lewald found that subjects consistently misreported sound locations as shifted in the direction of their own gaze [35]. These effects have been demonstrated only when gaze was strongly averted—at least 20°—whereas in our experiment, subjects fixated centrally. Nonetheless, we cannot exclude the possibility that a small overt shift in gaze direction contributed to the observed perceptual effect.
Because gaze and arrow cues are believed to be decoded through distinct and specialized neuronal pathways, it is striking that they evoke similar patterns of response including effects on both perceptual categorization and reaction time. Past research has shown that these two types of cues recruit distinct visual processing circuits. For example, one lesion study reported that split brain patients are cued by arrows in both hemifields, but by gaze only in the hemifield specialized for social processing [36]. Another found that a rare lesion damaging the right superior temporal gyrus eliminated gaze but not arrow cuing [37]. Nevertheless, after the initial perceptual processing, it appears likely that both gaze and arrow cues exert their influences through similar pathways. Two functional magnetic resonance imaging studies have reported only subtle differences between the orienting networks cued by gaze and arrows, and these differences may be more quantitative than qualitative [38,39].
In summary, observed eye-gaze and arrow cues robustly influence perceptual categorization and speed of response to lateralized sounds. The effects are dissociable: reaction time changes are robust to response difficulty, consistent with a simple stimulus–response compatibility effect, while changes in perceptual report were strongest near the category boundary, consistent with a cue-induced shift in perception. Our findings suggest that primate brains exploit learned statistical contingencies to better integrate directly and socially acquired knowledge, pooling information across conspecifics who, by virtue of shared evolutionary and developmental history, respond optimally to similar environmental affordances with similar behavioural responses [18]. Given that observed gaze direction is a ubiquitous signal in the lives of nonhuman primates, and that humans rapidly acquire experience with iconic cues, we have ample opportunities to observe the statistical validity of these social and symbolic cues. In conclusion, we report that social signals influencing attention also exert supramodal perceptual effects and thus synchronize our perceptions of our world.
Acknowledgements
Human participants provided informed consent under a protocol authorized by the Institutional Review Board of Princeton University and were debriefed at the conclusion of the session.
This work was supported by the Princeton University Office of the Dean of the College Senior Thesis Round Table Fund (J.I.B.), Princeton University Training Grant in Quantitative Neuroscience NRSA T32 MH065 214-1 (S.V.S.) and a National Science Foundation CAREER Award BCS-0547760 (A.A.G.). We thank Chandramouli Chandrasekaran for his programming assistance.
References
- 1.Driver J., Noesselt T. 2008. Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57, 11–23 10.1016/j.neuron.2007.12.013 (doi:10.1016/j.neuron.2007.12.013) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Ghazanfar A. A., Schroeder C. E. 2006. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 10.1016/j.tics.2006.04.008 (doi:10.1016/j.tics.2006.04.008) [DOI] [PubMed] [Google Scholar]
- 3.Ernst M. O., Buelthoff H. H. 2004. Merging the senses into a robust percept. Trends Cogn. Sci. 8, 162–169 10.1016/j.tics.2004.02.002 (doi:10.1016/j.tics.2004.02.002) [DOI] [PubMed] [Google Scholar]
- 4.Posner M. I. 1980. Orienting of attention. Q. J. Exp. Psychol. 32, 3–25 [DOI] [PubMed] [Google Scholar]
- 5.Kato M., Kashino M. 2001. Audio-visual link in auditory spatial discrimination. Acoust. Sci. Technol. 22, 380–382 10.1250/ast.22.380 (doi:10.1250/ast.22.380) [DOI] [Google Scholar]
- 6.Störmer V. S., McDonald J. J., Hillyard S. A. 2009. Cross-modal cueing of attention alters appearance and early cortical processing of visual stimuli. Proc. Natl Acad. Sci. USA 106, 22 456–22 461 10.1073/pnas.0907573106 (doi:10.1073/pnas.0907573106) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Spence C., McDonald J. 2004. The cross-modal consequences of the exogenous spatial orienting of attention. In The handbook of multisensory processes (eds Calvert G., Spence C., Stein B.), pp. 3–26 Cambridge, MA: MIT Press [Google Scholar]
- 8.Shepherd S. V. 2010. Following gaze: gaze-following behavior as a window into social cognition. Front. Iintegr. Neurosci. 4, 1–13 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Kuhn G., Kingstone A. 2009. Look away! Eyes and arrows engage oculomotor responses automatically. Attention Percep. Psychophys. 71, 314–327 10.3758/APP.71.2.314 (doi:10.3758/APP.71.2.314) [DOI] [PubMed] [Google Scholar]
- 10.Tipples J. 2002. Eye gaze is not unique: automatic orienting in response to uninformative arrows. Psychonom. Bull. Rev. 9, 314–318 [DOI] [PubMed] [Google Scholar]
- 11.Driver J., Davis G., Ricciardelli P., Kidd P., Maxwell E., Baron-Cohen S. 1999. Gaze perception triggers reflexive visuospatial orienting. Vis. Cogn. 6, 509–540 10.1080/135062899394920 (doi:10.1080/135062899394920) [DOI] [Google Scholar]
- 12.Friesen C. K., Kingstone A. 1998. The eyes have it! Relexive orienting is triggered by nonpredictive gaze. Psychonom. Bull. Rev. 5, 490–495 [Google Scholar]
- 13.Middlebrooks J. C., Green D. M. 1991. Sound localization by human listeners. Annu. Rev. Psychol. 42, 135–159 10.1146/annurev.ps.42.020191.001031 (doi:10.1146/annurev.ps.42.020191.001031) [DOI] [PubMed] [Google Scholar]
- 14.Wightman F. L., Kistler D. J. 1989. Headphones simulation of free-field listening. II. Psychophysical validation. J. Acoust. Soc. Am. 85, 868–878 10.1121/1.397558 (doi:10.1121/1.397558) [DOI] [PubMed] [Google Scholar]
- 15.Wichmann F. A., Hill N. J. 2001. The psychometric function. I. Fitting, sampling, and goodness of fit. Percept. Psychophys. 63, 1293–1313 [DOI] [PubMed] [Google Scholar]
- 16.Newport R., Howarth S. 2009. Social gaze cueing to auditory locations. Q. J. Exp. Psychol. 62, 625–634 10.1080/17470210802486027 (doi:10.1080/17470210802486027) [DOI] [PubMed] [Google Scholar]
- 17.Deaner R. O., Platt M. L. 2003. Reflexive social attention in monkeys and humans. Curr. Biol. 13, 1609–1613 10.1016/j.cub.2003.08.025 (doi:10.1016/j.cub.2003.08.025) [DOI] [PubMed] [Google Scholar]
- 18.Shepherd S. V., Steckenfinger S. A., Hasson U., Ghazanfar A. A. 2010. Human–monkey gaze correlations reveal convergent and divergent patterns of movie viewing. Curr. Biol. 20, 649–656 10.1016/j.cub.2010.02.032 (doi:10.1016/j.cub.2010.02.032) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Blakemore S. J., Frith C. 2005. The role of motor contagion in the prediction of action. Neuropsychologia 43, 260–267 10.1016/j.neuropsychologia.2004.11.012 (doi:10.1016/j.neuropsychologia.2004.11.012) [DOI] [PubMed] [Google Scholar]
- 20.Hatfield E., Cacioppo J. T., Rapson R. L. 1993. Emotional contagion. Curr. Directions Psychol. Sci. 2, 96–99 10.1111/1467-8721.ep10770953 (doi:10.1111/1467-8721.ep10770953) [DOI] [Google Scholar]
- 21.Willis J., Todorov A. 2006. First impressions: making up your mind after a 100-ms exposure to a face. Psychol. Sci. 17, 592–598 10.1111/j.1467-9280.2006.01750.x (doi:10.1111/j.1467-9280.2006.01750.x) [DOI] [PubMed] [Google Scholar]
- 22.Adams R. B., Kleck R. E. 2005. Effects of direct and averted gaze on the perception of facially communicated emotions. Emotion 5, 3–11 10.1037/1528-3542.5.1.3 (doi:10.1037/1528-3542.5.1.3) [DOI] [PubMed] [Google Scholar]
- 23.Bayliss A. P., Frischen A., Fenske M. J., Tipper S. P. 2007. Affective evaluations of objects are influenced by observed gaze direction and emotional expression. Cognition 104, 644–653 10.1016/j.cognition.2006.07.012 (doi:10.1016/j.cognition.2006.07.012) [DOI] [PubMed] [Google Scholar]
- 24.Hietanen J. K., Leppanen J. M. 2003. Does facial expression affect attention orienting by gaze direction cues? J. Exp. Psychol. Hum. Percept. Perform. 29, 1228–1242 [DOI] [PubMed] [Google Scholar]
- 25.Mathews A., Fox E., Yiend J., Calder A. 2003. The face of fear: effects of eye gaze and emotion on visual attention. Vis. Cogn. 10, 823–835 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Graham R., Friesen C. K., Fichtenholtz H. M., LaBar K. S. 2010. Modulation of reflexive orienting to gaze direction by facial expressions. Vis. Cogn. 18, 331–368 10.1080/13506280802689281 (doi:10.1080/13506280802689281) [DOI] [Google Scholar]
- 27.Ghazanfar A. A., Santos L. R. 2004. Primate brains in the wild: the sensory bases for social interactions. Nat. Rev. Neurosci. 5, 603–616 10.1038/nrn1473 (doi:10.1038/nrn1473) [DOI] [PubMed] [Google Scholar]
- 28.Platt M. L., Ghazanfar A. A. 2010. Primate neuroethology. Oxford, UK: Oxford University Press [Google Scholar]
- 29.Buchtel H. A., Butter C. M. 1988. Spatial attention shifts: implications for the role of polysensory mechanisms. Neuropsychologia 26, 499–509 10.1016/0028-3932(88)90107-8 (doi:10.1016/0028-3932(88)90107-8) [DOI] [PubMed] [Google Scholar]
- 30.Schmitt M., Postma A., De Haan E. 2000. Interactions between exogenous auditory and visual spatial attention. Q. J. Exp. Psychol. 53A, 105–130 [DOI] [PubMed] [Google Scholar]
- 31.Gopher D. 1973. Eye-movement patterns in selective listening tasks of focused attention. Percept. Psychophys. 14, 259–264 [Google Scholar]
- 32.Rorden C., Driver J. 1999. Does auditory attention shift in the direction of an upcoming saccade? Neuropsychologia 37, 357–377 10.1016/S0028-3932(98)00072-4 (doi:10.1016/S0028-3932(98)00072-4) [DOI] [PubMed] [Google Scholar]
- 33.Ringo J. L., Sobotka S., Diltz M. D., Bunce C. M. 1994. Eye-movements modulate activity in hippocampal, parahippocampal, and inferotemporal neurons. J. Neurophysiol. 71, 1285–1288 [DOI] [PubMed] [Google Scholar]
- 34.Werner-Reiss U., Kelly K. A., Trause A. S., Underhill A. M., Groh J. M. 2003. Eye position affects activity in primary auditory cortex of primates. Curr. Biol. 13, 554–562 10.1016/S0960-9822(03)00168-4 (doi:10.1016/S0960-9822(03)00168-4) [DOI] [PubMed] [Google Scholar]
- 35.Lewald J. 1997. Eye-position effects in directional hearing. Brain Res. 87, 35–48 [DOI] [PubMed] [Google Scholar]
- 36.Kingstone A., Friesen C. K., Gazzaniga M. S. 2000. Reflexive joint attention depends on lateralized cortical connections. Psychol. Sci. 11, 159–166 10.1111/1467-9280.00232 (doi:10.1111/1467-9280.00232) [DOI] [PubMed] [Google Scholar]
- 37.Akiyama T., Kato M., Muramatsu T., Saito F., Nakachi R., Kashima H. 2006. A deficit in discriminating gaze direction in a case of right superior temporal gyrus lesion. Neuropsychologia 44, 161–170 10.1016/j.neuropsychologia.2005.05.018 (doi:10.1016/j.neuropsychologia.2005.05.018) [DOI] [PubMed] [Google Scholar]
- 38.Hietanen J. K., Nummenmaa L., Nyman M. J., Parkkola R., Hamalainen H. 2006. Automatic attention orienting by social and symbolic cues activates different neural networks: an fMRI study. NeuroImage 33, 406–413 10.1016/j.neuroimage.2006.06.048 (doi:10.1016/j.neuroimage.2006.06.048) [DOI] [PubMed] [Google Scholar]
- 39.Tipper C. M., Handy T. C., Giesbrecht B., Kingstone A. 2008. Brain responses to biological relevance. J. Cogn. Neurosci. 20, 879–891 10.1162/jocn.2008.20510 (doi:10.1162/jocn.2008.20510) [DOI] [PubMed] [Google Scholar]




