Skip to main content
NCBI home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2012 Feb 8;32(6):2159–2165. doi: 10.1523/JNEUROSCI.5094-11.2012

Hemispheric Asymmetry and Visuo-Olfactory Integration in Perceiving Subthreshold (Micro) Fearful Expressions

Emily C Forscher 1, Wen Li 1,2,✉
PMCID: PMC3568979  NIHMSID: NIHMS439345  PMID: 22323728

Abstract

Multisensory integration is ubiquitous, facilitating perception beyond the limit of individual senses. This mechanism is especially salient when individual sensory input is weak (i.e., the principle of inverse effectiveness), fusing subthreshold cues into tangible percepts. Nevertheless, it is unclear how this rule applies to threat perception, synthesizing elusive, discrete traces of a threat into a discernible danger signal. In light of hemispheric asymmetry in threat processing, we combined parafoveal stimulus presentation and the contralateral P1 visual event-related potential to investigate how aversive olfactory inputs enhance visual perception of highly degraded, subthreshold fearful expressions. The dominant right hemisphere exhibited early visual discrimination between subtle fear and neutral expressions, independently of accompanying odors. In the left hemisphere, differential visual processing occurred only at the convergence of negative odors and minute facial fear, highlighting the success and necessity of visuo-olfactory threat integration in this disadvantaged hemisphere. Reaction time data from a subsequent dot-detection task complemented these neural findings, revealing odor-dependent and hemisphere-specific modulation of spatial attention to facial expressions. Our evidence thus indicates cross-modal threat integration in basic visual perception in humans that captures minimal threat information, especially in the blind right hemifield. Critically, this interaction between multisensory synergy and hemispheric asymmetry in threat perception may underlie the multifaceted fear experiences of everyday life.

Introduction

Efficient detection of threat in the biological landscape often means life or death for an organism. All species equipped with multiple senses integrate multimodal inputs to optimize perception (Stein and Meredith, 1993); this mechanism is especially prominent when scarce (rather than rich) information is available to individual senses [the principle of inverse effectiveness (Stein and Meredith, 1993; Guo and Guo, 2005)]. Accordingly, cross-modal integration would make vast ecological sense by fusing weak, discrete traces (e.g., a faint smell, a fast-moving object) into a discernible danger signal (e.g., a hunting tiger), promptly initiating adaptive responses to avert or overcome threat.

Micro facial expressions (e.g., a concealed/suppressed fear expression) are typically undetectable to the human eye without systemic training (Ekman, 2003), but nevertheless bear great biological significance. Preliminary behavioral evidence suggests that olfactory input can assist emotion detection in ambiguous faces (Zhou and Chen, 2009), and can occur outside of conscious awareness (Li et al., 2007b), raising the possibility of mandatory visuo-olfactory integration in the recognition of micro facial expressions. Indeed, olfactory input, among all senses, could be especially potent in multimodal emotion integration, given the tendency of smells to induce strong hedonic responses (Schiffman, 1974; Yeshurun and Sobel, 2010) and to heighten arousal and attention (Miltner et al., 1994; Ehrlichman et al., 1997), and given the intimate anatomical connection between the olfactory system and emotion-related limbic regions (Carmichael et al., 1994; Gottfried and Zald, 2005; Zelano et al., 2005). Several studies combining visual and auditory emotional cues notwithstanding (de Gelder et al., 1999; Dolan et al., 2001; Pourtois et al., 2005), cross-modal synergy in processing subtle/subthreshold emotion remains obscure, and research is scant concerning visuo-olfactory emotion integration and its neural underpinnings.

Threat cues typically arise first in the periphery and then capture central vision by their biological salience. Peripheral vision preferentially recruits the fast (yet coarse) magnocellular visual pathway (Livingstone and Hubel, 1988), triggering “vision for action” to enable rapid threat detection and promote attention orientation and other adaptive responses (Milner and Goodale, 2008). However, peripheral visual acuity is generally compromised, especially when key content is subtle. Furthermore, as parafoveal stimulation activates the visual system in the contralateral hemisphere, in light of right-hemispheric dominance of emotion processing (Adolphs et al., 1996; Borod et al., 1998; Davidson and Irwin, 1999), threat detection in the nonpreferred right visual periphery is particularly challenging. Accordingly, complementary olfactory input could be critical by inducing multisensory integration to enhance threat perception in the disadvantaged hemisphere. Consistent with the rule of inverse effectiveness, the nondominant hemisphere exhibits superiority in standard (nonemotional) multisensory integration (Halsband and Hömberg, 1990), but little is known about how cross-modal integration interacts with hemispheric specialization in threat processing.

Using the P1 visual event-related potential (ERP), we examined how olfactory input facilitates basic visual processing of subthreshold fearful (micro fear) expressions. P1 is a well established primary visual ERP generated in the extrastriate cortex ∼100 ms poststimulus, which reliably reflects early visual processing and is sensitive to emotional modulation of visual perception and attention (Mangun et al., 1993; Eimer and Holmes, 2007; Vuilleumier and Pourtois, 2007; Li et al., 2008a). Parsing early visual analysis in different hemispheres, parafoveal face presentation and contralateral P1 provided a critical means to examine hemispheric asymmetry in cross-modal threat integration. We also extracted the N170 component that is sensitive to configural face processing (Bentin et al., 1996) to explore olfactory–visual interaction in later visual analysis. Following face presentation, a subsequent dot-probe detection task demonstrated how this perceptual mechanism translated into spatial attention and overt behavior. Finally, a control ERP experiment was conducted, which aimed to rule out confounds related to low-level physical properties of the pictures.

Materials and Methods

Participants

Forty-three undergraduates at the University of Wisconsin–Madison (20 female; mean age, 19.0 years) participated in exchange for class credit in an introductory psychology course. All participants were right-handed, with normal olfaction and normal or corrected-to-normal vision. Five participants with excessive EEG interference were excluded from ERP analyses, but remained in behavioral analyses. All participants provided informed consent, which was approved by the University of Wisconsin Institutional Review Board.

Anxiety assessment

As anxiety is associated with heightened sensitivity to threat (Mathews and MacLeod, 2005), we measured individual levels of anxiety to assist our elucidation of cross-modal threat integration. At the beginning of the experiment, each participant completed two self-report inventories. The Beck Anxiety Inventory (BAI) (Beck and Steer, 1993) consists of 21 items assessing the severity of anxiety symptoms and is commonly used in clinical and research settings. The Behavioral Inhibition Scale (BIS) (Carver and White, 1994) consists of seven items assessing a personality trait related to proneness to general anxiety (i.e., trait anxiety) (Li et al., 2007a, 2008a). Both questionnaires are rated on a 4-point scale between 0 and 3 (for the BAI) or between 1 and 4 (for the BIS), ranging from “very little” to “very much”. BIS and BAI scores were closely correlated (r = 0.47, p = 0.001). Given this high convergent validity between the two scales, which exhibited largely similar modulatory effects on threat processing, and given the close association between trait anxiety and anxiety symptoms (Johnson et al., 2003; Campbell-Sills et al., 2004), BIS and BAI scores were standardized and averaged to form an anxiety composite to reflect general anxiety (Krusemark and Li, 2011). We note that results related to individual indices (reported parenthetically) were closely aligned to those involving the composite index.

Stimuli

Neutral and micro fear face stimuli.

Pictures of 16 models (8 female) expressing fearful and neutral expressions were selected from the Karolinska Directed Emotional Faces (KDEF) (Lundqvist et al., 1998). All images were frontal views, in color, with a consistent background. Fearful and neutral pictures of each model were then morphed together using FantaMorph (Abrosoft) on a continuum of 0% (pure neutral) to 100% (pure fearful) to create graded fearful expressions.

We first estimated the fear detection threshold for these morphs in an independent sample (N = 12), who viewed a 6 s video playback of each facial morph and responded via button press upon detection of fear in the video. Participants reported fear detection at 30–42% (mean 36%) for the 16 morphs, varying with each model's fear expressiveness. To ensure that fearful expressions in the experiment were below detection, we set the final morph percentage at 35% of the above fear detection threshold for each morph, resulting in a set of micro fear face stimuli containing, on average, 12.5% of the full fear expression (ranging from 10% to 15% among the models). Neutral face stimuli were created using a 2% morph on the neutral-to-fear continuum so that they generally matched the micro fear face images in visual alterations caused by the morphing procedure (Fig. 1A).

Figure 1.

Figure 1.

Open in a new tab

Experiment paradigm. A, Example face pair used in the main experiment. All images were presented in color. B, A typical trial in the main experiment. C, Three psychophysical posttests and respective group-level accuracy results.

Olfactory stimuli.

Rose oxide (grass, medicine; 25% dilution in mineral oil) and α-pinene (pine resin; 40%) were chosen as neutral odors, and valeric acid (sweat/rotten cheese; 5%) and hexanoic acid (rotten meat/fat; 10%) as negative odors. Before the experiment, participants provided valence ratings for each odorant on a visual analog scale from −10 (extremely unpleasant) to 10 (extremely pleasant). Ratings confirmed that negative odors were significantly more unpleasant than neutral odors [mean (SD) = −4.88 (2.72) and −0.98 (3.44), respectively; t(44) = −10.23, p < 0.001]. We also obtained ratings of odor intensity, familiarity, and pungency on a similar scale (−10 to 10), which together formed an olfactory composite measure. We entered this composite index as a covariate into the statistical analyses described below to partial out nonemotional olfactory variations.

Procedure

Participants were seated in a dimly lit, sound-attenuated room, 120 cm from a 20-inch CRT monitor. They completed a dot-probe task, followed by a set of psychophysical tests assessing fear awareness and other potential confounds (see Posttests, below).

Main experiment: dot-probe task.

As illustrated in Figure 1B, each trial began with a “Sniff Now” cue and 800 ms of odor presentation, followed by a face pair displayed for 250 ms. After face offset, a dot probe was displayed at the location of one of the faces for 500 ms. While maintaining central fixation, participants were asked to respond via button press whether the dot appeared on the left or right side of the screen. To encourage central fixation, a fixation cross remained onscreen throughout. The intertrial interval was set at 11 s to prevent olfactory habituation and fatigue. Each face pair consisted of a neutral face and a micro fear face of the same model, with either face type appearing in the left or right visual field (L/RVF) equally frequently. Each face subtended a visual area of 2.4° by 3.2°, with a horizontal eccentricity of 7.1° to ensure truly parafoveal vision. A total of 192 trials were randomly presented in three blocks of 64 trials each. The first eight trials of the experiment were practice trials and thus excluded from analysis. All conditions were counterbalanced across trials. Stimulus presentation and response collection were controlled using Cogent2000 software (Wellcome Department, London, UK), as implemented in Matlab (Mathworks). Odorants were delivered using an eight-channel computer-controlled olfactometer (airflow set at 1.5 L/min), which permits rapid delivery of odor in the absence of tactile, thermal, or auditory confounds (Li et al., 2006, 2008b).

Posttests.

After the dot-probe task, we probed for awareness of differences within face pairs, using a brief funnel interview with three specific questions in sequence: (1) Did you see any differences between the two faces in each trial? (2) What kind of differences? (3) How sure are you about what you saw? None of the participants reported noticing any differences between the two faces in each pair, thereby excluding subjective awareness of fear expressions.

We then administered three forced-choice psychophysical tests. The first two tasks were designed to objectively assess participants' awareness of fear in the faces (Fig. 1C). In Task 1 (central fear recognition task), we presented a single face picture centrally, to which participants indicated whether it contained fear. In Task 2 (peripheral fear discrimination task), we presented the original 16 face pairs in the same manner as the main experiment, wherein participants indicated which side contained a micro fear face. Task 3 (physical discrimination task) was designed to exclude visible physical discrepancies in the face pair that could drive differential visual processing of the faces. To that end, we presented each neutral face picture with a morphed image between the neutral picture and another model's neutral image at the same morph percentage as the micro fear images. As such, the two faces contained equal physical disparities to the neutral-micro-fear pair, but were both affectively neutral. Participants responded whether the two faces were the same or different. In each task, there were 32 trials, with each face displayed for the same duration and with the same visual angle and eccentricity (in Tasks 2 and 3) as the main experiment.

EEG data acquisition and analysis

EEG was recorded throughout the main experiment from a 96-channel (ActiveTwo; BioSemi) system at a 1024 Hz sampling rate with a 0.1–100 Hz bandpass filter. Electrooculogram (EOG) was recorded at two eye electrodes at the outer canthi and one infraorbital to the left eye. EEG signals were referenced offline to the average of the left and right mastoid recordings. Horizontal EOG channels were referenced to each other, and the vertical EOG channel was referenced to the EEG channel located directly above it. EEG/EOG signals were digital bandpass filtered from 0.1 to 40 Hz, down-sampled to 256 Hz, segmented around face onset (from −200 to 300 ms), and corrected to a 200 ms prestimulus baseline. Trials with EEG/EOG voltages exceeding ±75 μV were excluded from analysis (Krusemark and Li, 2011).

Inspection of the grand average ERP waveform confirmed a marked P1 component at occipital sites, peaking at 100 ms after face onset. Importantly, as P1 is selective to contralateral (vs ipsilateral) visual stimulation and the faces were presented with significant eccentricity in the LVF and RVF, we used contralateral P1s to index visual responses to faces in the hemifields. That is, P1 at O1 would index response to the face in the RVF, and P1 at O2 to the face in the LVF, constituting two P1 measures in each trial (Fig. 2A). These P1 indices were quantified by mean amplitudes extracted over a 44 ms interval, centering on the peak (±5 data points between 76 and 120 ms, corrected to 200 ms prestimulus baseline). Maximal N170 was located at left and right lateral occipitoparietal sites: PO7 and PO8. In a similar manner to P1 analysis, we extracted N170 mean amplitudes (44 ms between 140 and 184 ms, centering on the N170 peak) corresponding to contralateral visual fields.

Figure 2.

Figure 2.

Open in a new tab

Occipital P1 revealed visual and olfactory–visual capture of micro fearful expressions. A, Diagram illustrating contralateral correspondence between parafoveal visual presentation and hemispheric P1 recording site. B, ERP waveforms at right occipital site O2 in response to LVF faces. C, Scalp map of differential P1 (micro fear—neutral) to LVF faces; four sensors bolded and circled to represent O2. D, ERP waveforms at left occipital site O1 in response to RVF faces in the negative olfactory context in high and low anxiety. E, Scalp maps of differential P1 (micro fear—neutral) to RVF faces in high and low anxiety groups. Four sensors bolded and circled to represent O1. LH, Left hemisphere; RH, right hemisphere; contra., contralateral; Anx., anxiety.

Statistical analysis

Given a priori hemispheric asymmetry in face and emotion processing discussed above, analysis was conducted in RVF/left hemisphere and LVF/right hemisphere separately to maximize statistical power. Mean contralateral P1/N170 amplitudes and reaction time (RT) to the dot probe were separately entered into three-way repeated-measures ANOVAs, which included odor valence (neutral/negative), face valence (neutral/micro fear), and anxiety (a continuous variable) as independent variables. Importantly, odor composite scores were included as a covariate whenever odor valence (main or interactive) effects were analyzed. Incorrect dot detection responses (0.4% on average) or RTs beyond ±2 SD from the participant's mean RT were excluded. Accuracies in posttasks were submitted to t tests and binomial tests.

Results

Psychophysical posttests

As shown in Figure 1C, mean accuracy in all three tasks was at chance level (0.50), as confirmed by t tests (ps > 0.50). Additionally, in neither Task 2 nor Task 3 did performance vary as a function of whether micro fear faces were displayed in the LVF or RVF (ps > 0.54). At the individual level, binomial tests indicated that not a single participant performed above chance (ts < 0.68, ps > 0.50). We also note that anxiety composite (and individual BIS and BAI) scores did not correlate with accuracy in any of the tasks (ps > 0.35), suggesting that the fearful expressions appeared equally weak to high- and low-anxious individuals. To summarize, the critical Task 2 (simulating visual stimulation in the main experiment), along with Task 1 (assessing central fear recognition), validated the manipulation of subthreshold fear presentation. Additionally, failure to differentiate neutral faces from their nonemotional morphs in Task 3 further suggested that physical disparities induced by morphing at the applied percentage could not support visual discrimination.

Dot-detection RT

Using a dot-probe detection task following face presentation, we derived a behavioral index of selective spatial attention to fearful versus neutral faces. Differential RT to dots replacing the two types of faces would support selective spatial attention and, by extension, reflect emotion discrimination. In keeping with the right-hemisphere dominance theory, when the dot appeared in the corresponding LVF, we observed an interaction between face valence and odor valence (F(1,39) = 4.32, p < 0.05) driven by a significant effect of face valence in the context of negative odors (t(42) = 2.89, p < 0.01; Fig. 3). RT for micro fear face trials [375 ms (77)] was significantly slower than RT for neutral trials [368 ms (71)]. In contrast, when accompanied by neutral odors, face valence effect was negligible (t(42) = 0.03, p = 0.98). Finally, with the dot in the nondominant RVF, there was no effect of face valence alone or interactively with other factors (ps > 0.35).

Figure 3.

Figure 3.

Open in a new tab

Reaction time in the dot-probe task. *p < 0.05, **p < 0.005; Error bars indicate individual-mean-adjusted SEM.

P1 response

First, we observed a differential P1 between micro fear and neutral faces at (right occipital) site O2 (corresponding to the LVF; Fig. 2B,C). A three-way ANOVA confirmed a main effect of face valence [F(1,34) = 5.28, p < 0.05] independent of any other factors (ps > 0.1). Micro fear faces augmented P1 relative to neutral faces [mean (SD) = 2.35 μV (2.17) and 1.97 μV (2.50), respectively]. Concerning fear processing in the nondominant left hemisphere, a similar ANOVA on P1 at (left occipital) site O1 yielded a single significant effect, which was a three-way interaction of face valence, odor valence, and anxiety (composite scores: F(1,34) = 4.97, p < 0.05; when anxiety was indexed by the BIS or BAI individually: F = 3.50 or 3.57, p = 0.07 or 0.07, respectively). Breaking this interaction down by paired odors, we observed a trend-level effect of face valence (F(1,36) = 3.24, p = 0.08) and a trend-level interaction between face valence and anxiety (F(1,36) = 3.42, p = 0.07; when anxiety was indexed by the BIS or BAI individually: F = 5.11 or 1.81, p = 0.03 or 0.18, respectively), both in the presence of negative odors alone. As Figure 2, D and E, illustrates, the high-anxiety group (by median split) discriminated between the two types of faces in the presence of negative odors (albeit the divergence appeared in the opposite direction to the right occipital P1 effect above): P1 was augmented to neutral faces [2.05 μV (2.93)] relative to micro fear faces [1.19 μV (2.82); t(19) = −2.25, p < 0.05; when anxiety was indexed by the BIS or BAI: t = −2.25 or −1.86, p = 0.04 or 0.08, respectively]. In contrast, the low-anxiety group exhibited overlapping P1 waveforms for the two conditions (p = 0.87). In the presence of neutral odors, no significant effects were observed (ps > 0.39).

N170 response

An analogous set of analyses on N170 mean amplitudes revealed a main effect of odor at both PO7 and PO8 (F(1,35) > 4.72, ps < 0.05), while face valence failed to reach the level of significance as a main or interactive effect (ps > 0.1). N170 was attenuated by negative versus neutral odors [PO7: −1.48 μV (2.55) vs −1.95 μV (2.46); PO8: −1.70 μV (3.14) vs −2.17 μV (3.02), respectively], suggesting negative-odor-related interference in visual processing. Guided by the RT and P1 effects above, we conducted post hoc contrasts between micro fear and neutral faces in the presence of negative odors. At PO8 (corresponding to the LVF), we observed a trend-level decrease in N170 to micro fear versus neutral faces [−1.47 μV (3.24) vs −1.94 μV (3.23), respectively; t(37) = 1.87, p = 0.07]. While this differentiation of face affect aligned with the above RT effect in the LVF, the N170 at PO7 indicated no effect of face valence (p = 0.64).

Control experiment: control of low-level influences on early visual processing

To further rule out the possibility that differential P1/N170 effects could stem from minor physical differences between the two types of face stimuli, we conducted a control ERP experiment. We assessed P1/N170 responses to inverted versions of the neutral-and-micro-fear face pairs, which preserved the physical properties of the images while minimizing facial emotion processing.

Participants and procedure

Ten additional undergraduates at the University of Wisconsin–Madison (5 female; mean age, 18.8 years) performed the same dot-probe task as applied in the main experiment (with the same number of trials), except that there were no odor stimuli. Mean contralateral P1/N170 amplitudes were extracted as in the main experiment and then submitted into a repeated-measures ANOVA with visual field and face valence as independent variables, and anxiety level as a continuous covariate.

P1/N170 response to inverted faces

In contrast to the main experiment, there was not any significant P1 effect of facial emotion, either independently or interactively with other factors (F(1,8)s < 1.13, ps > 0.31). Similarly, no effects related to face valence on N170 amplitude reached the level of significance (F(1,8)s < 1.15, ps > 0.31). Furthermore, given that all the effect sizes were small (η2s < 0.12) and would not reach the level of significance even with the same sample size as the main study (N = 38), these supplemental results ruled out inherent physical differences between the two face valence types that could potentially drive the P1 effects reported above.

Discussion

By isolating parafoveal fearful expressions that were highly degraded and consciously undetectable, the right-hemispheric visual system exhibited remarkable acuity in threat perception. A still greater feat of human threat perception emerged in the nondominant left hemisphere, which captured this minute amount of fear by integrating affective signals from the olfactory sense, especially among anxious participants. Furthermore, in the preferred LVF and in the presence of negative odors, these micro fear expressions shifted spatial attention in the subsequent dot-detection task. Therefore, beyond replicating right-hemispheric specialization in threat processing, these findings afford some of the first evidence of visuo-olfactory synergy in threat perception, to the extent that it defies the usual blindness of the human eye to elusive facial emotions. Importantly, cross-modal threat integration interacts with hemispheric asymmetry, thereby flexibly modulating the direction of threat processing (i.e., approach vs avoidance).

A series of psychophysical tests at the end of the study systemically assessed the level of awareness of facial fear and validated that the micro fear emotion was subthreshold by both subjective and objective standards (Hannula et al., 2005). By randomly jittering the presentation of odor and face stimuli to dissociate their respective electrical responses, we minimized carry-over electrical signals related to olfactory stimuli that could confound the P1 effects. Finally, the control ERP experiment using inverted face pairs rejected the account that the reported P1/N170 effects were mediated by disparate physical properties of the two types of faces. Moreover, the synergy among micro fear faces, negative odors, and anxiety further implicates an affect-based mechanism at play here.

The fact that subthreshold parafoveal fear in the LVF independently evokes preferential visual response as early as 100 ms (as indexed by the right occipital P1) replicates previous studies evaluating subliminal threat perception in early stages (Liddell et al., 2004; Williams et al., 2004; Li et al., 2008a; Bayle et al., 2009). However, those studies all used full emotional expressions rendered consciously imperceptible with suppression techniques, such as masking. Therefore, beyond general unconscious processing of threat, the current finding highlights the capacity of the visual system in the right hemisphere to register extremely minor disparities in emotional content (2% vs 12.5% fear). In fact, the right visual system is so sensitive to aversive threat cues that cross-modal integration would not afford incremental information to visual processing here. Given that the magnocellular visual pathway is selectively recruited by the parafoveal stimuli here, such fine visual discrimination further accentuates the possibility that this purportedly coarse visual route is actually capable of refined stimulus analysis, at least in the emotion domain.

As we predicted, left occipital P1, by contrast, revealed successful visuo-olfactory threat integration in the left visual system, which was necessary for distinguishing RVF micro fear faces from their neutral counterparts. Notably, this effect is primarily evident in anxious individuals characterized by heightened threat sensitivity (Mathews and MacLeod, 2005). Such cross-modal integration of threat in the disadvantaged left hemisphere coincides with the principle of inverse effectiveness (Stein and Meredith, 1993; Guo and Guo, 2005) and the superiority of the nondominant hemisphere in standard multimodal integration (Halsband and Hömberg, 1990). As such, the interaction between hemispheric asymmetry and cross-modal integration in threat perception may serve a critical ecological function by specifically boosting threat analysis that is otherwise largely limited. Interestingly, to the extent that the P1 modulation reflected emotion-related early visual sensory perception and attention, the N170 here exhibited a general reduction caused by negative odors, implicating nonselective interference in later/higher-order visual processing due to competing aversive olfactory processing. This finding conforms to the view of N170 being associated with standard configural encoding of facial features rather than with emotional significance specifically (for review, see Vuilleumier and Pourtois, 2007).

The rapid rise of signal binding indexed by P1 accentuates multisensory integration in early sensory processing. The fact that it took place when fear information was subliminal and olfactory input incidental further emphasizes the automatic, low-level nature of cross-modal integration (de Gelder et al., 1999; Dolan et al., 2001), as opposed to representing a product of higher-order cognitive synthesis (Chen et al., 2010; Klasen et al., 2011). Several neuroimaging studies have investigated audiovisual emotion integration in the analysis of suprathreshold and foveal visual cues, which implicated the amygdala as a key site where multimodal affective information merges (Dolan et al., 2001; Pourtois et al., 2005; Klasen et al., 2011). Indeed, the extensive anatomical connectivity between the amygdala and every sensory cortex (Turner et al., 1980; Modha and Singh, 2010) can provide the necessary architecture for the amygdala to unify multisensory emotional signals and send feedback projections to corresponding sensory cortices to enhance perception. Nevertheless, the amygdala response to emotion is known to emerge after 100 ms (Oya et al., 2002; Krolak-Salmon et al., 2004), which virtually eliminates the possibility of amygdala-reentrant input driving the fast P1 effect observed here. One possible explanation is that the magnocellular visual pathway (preferentially tuned to the parafoveal stimuli in this experiment) supports swift transmission of the fear signal to the amygdala, resulting in a saving of >20 ms compared with parvocellular transmission of foveal input (Bullier, 2001) and thus permitting the amygdala projections to contribute to early sensory integration.

Alternatively, we propose that multisensory neurons in the superior colliculus, a key structure in standard cross-modal integration (Stein and Stanford, 2008), may mediate this multisensory integration of emotion. As peripheral fear input arrives at the superior colliculus via the magnocellular pathway (retina→superior colliculus→pulvinar), it merges with olfactory information in the multisensory neurons there, projecting integrated neural impulses onward to the associate visual cortex (e.g., the extrastriate cortex) (Bullier, 2001). In light of accruing evidence of independent threat encoding in the (associate) sensory cortex (Li et al., 2008b; Pessoa and Adolphs, 2010; Sacco and Sacchetti, 2010), we reason that these synthesized (as opposed to unimodal visual) afferents from the superior colliculus generate sufficient neuronal firing to activate fear representations stored in the visual cortex, thereby underlying this early odor-dependent P1 isolation of minimal fear in the left hemisphere.

Merging olfactory and visual threat cues not only allows for sensory discrimination of threat, but may modify the direction of threat processing, aligning it to the ongoing motivational need. Visuo-olfactory threat integration in this study manifested in the form of suppressed visual processing (reduced P1) and averted spatial attention (slowed dot detection), as opposed to response facilitation usually induced by threat. Critically, unlike typical dot-detection paradigms, a negative odor preceded threat presentation and dot detection in the current study, and only within this negative olfactory context did we observe (both behaviorally and neurally) these inhibitory responses to facial fear. These response profiles appear in keeping with vigilance–avoidance hypotheses of threat processing: following initial automatic vigilance to the signal, threat induces cognitive and behavioral avoidance, especially in anxiety (Mogg et al., 2004; Weinberg and Hajcak, 2011). That is, aversive odors (the first event in a given trial here) could evoke initial emotional and sensory vigilance (e.g., in the amygdala and upstream sensory structures), which then sets off top-down inhibitory projections (e.g., from the prefrontal cortex) to the visual brain, where a series of visual inputs (i.e., faces and the dot probe) successively arrive (Fenske et al., 2006), exerting visual suppression and attentional avoidance in threat processing. Therefore, rather than simply enhancing threat perception, cross-modal threat integration may pose broader biological implications, by flexibly guiding sensory perception and subsequent operations according to different motivational situations.

Despite reliable encoding of hedonicity in early visual processing (based on unimodal or bimodal cues), our behavioral data suggest that this operation does not directly translate into corresponding behavior. Fear encoding in the nondominant left hemisphere did not yield differential performance in the subsequent dot detection task, regardless of odor context. Additionally, failed fear discrimination, even in the preferred LVF (Post Task 2), indicates that efficient right-hemispheric visual encoding of threat is not sufficient for behavioral discrimination of minimal threat. Instead, as the dot detection data indicated, a congruent olfactory context is required for explicit behavioral consequences of LVF micro fear faces to transpire. Notably, this behavioral outcome coincided with possible threat differentiation in the right hemisphere N170 component, highlighting the idea that negative olfactory context facilitated later visual discrimination of LVF micro fear and neutral faces. These intricate neurobehavioral dynamics dovetail with neural models that emphasize an internal threshold and accumulative information processing in the production of behavioral responses (Lamme and Roelfsema, 2000; Dehaene et al., 2006; Ratcliff and McKoon, 2008). That is, early visual response evoked by minimal threat in and of itself may not exceed a given threshold. However, when integrated with additional processes (e.g., olfactory analysis), especially in the dominant right hemisphere, it may overcome the threshold to trigger and sustain downstream operations (e.g., possible N170 discrimination of LVF threat), culminating in overt behavioral effects (e.g., differential RT to LVF threat). Therefore, despite its high biological salience, an internal threshold could still be preserved in threat processing, which may serve to spare the body from a costly full-blown emotional reaction in the face of minor, isolated danger cues.

To the extent that visual perception dominates human sensory experiences, this ability is finite and constrains what people actually see. Subtle, fleeting facial expressions often evade conscious perception, sometimes at a great cost to the viewer. Here, we demonstrate an extraordinary feat of human vision in catching elusive traces of fearful expressions, even in the nonpreferred right visual space, by efficiently melding olfactory affective information into visual processing of faces. Critically, cross-modal synthesis of threat does not monotonically improve emotion perception. Rather, it modulates information processing and motivation direction—a mechanism potentially responsible for the multifaceted and sometimes paradoxical fear responses we experience in everyday life.

Footnotes

We thank Drs. Richard Davidson, Joseph Newman, and Elizabeth Krusemark for thoughtful comments on the manuscript. We also thank Lucas Novak for assistance with data collection.

This study was supported by a core grant to the Waisman Center from the NICHD (P30 HD03352).

The authors declare no competing financial interests.

References

  1. Adolphs R, Damasio H, Tranel D, Damasio AR. Cortical systems for the recognition of emotion in facial expressions. J Neurosci. 1996;16:7678–7687. doi: 10.1523/JNEUROSCI.16-23-07678.1996. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Bayle DJ, Henaff MA, Krolak-Salmon P. Unconsciously perceived fear in peripheral vision alerts the limbic system: a MEG study. PLoS One. 2009;4:e8207. doi: 10.1371/journal.pone.0008207. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Beck AT, Steer RA. Beck anxiety inventory manual. San Antonio, TX: Psychological Corporation; 1993. [Google Scholar]
  4. Bentin S, Allison T, Puce A, Perez E, McCarthy G. Electrophysiological studies of face perception in humans. J Cogn Neurosci. 1996;8:551–565. doi: 10.1162/jocn.1996.8.6.551. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Borod JC, Cicero BA, Obler LK, Welkowitz J, Erhan HM, Santschi C, Grunwald IS, Agosti RM, Whalen JR. Right hemisphere emotional perception: evidence across multiple channels. Neuropsychology. 1998;12:446–458. doi: 10.1037//0894-4105.12.3.446. [DOI] [PubMed] [Google Scholar]
  6. Bullier J. Integrated model of visual processing. Brain Res Rev. 2001;36:96–107. doi: 10.1016/s0165-0173(01)00085-6. [DOI] [PubMed] [Google Scholar]
  7. Campbell-Sills L, Liverant GI, Brown TA. Psychometric evaluation of the behavioral inhibition/behavioral activation scales in a large sample of outpatients with anxiety and mood disorders. Psychol Assess. 2004;16:244–254. doi: 10.1037/1040-3590.16.3.244. [DOI] [PubMed] [Google Scholar]
  8. Carmichael ST, Clugnet MC, Price JL. Central olfactory connections in the macaque monkey. J Comp Neurol. 1994;346:403–434. doi: 10.1002/cne.903460306. [DOI] [PubMed] [Google Scholar]
  9. Carver CS, White TL. Behavioral-inhibition, behavioral activation, and affective responses to impending reward and punishment: the BIS/BAS scales. J Person Soc Psychol. 1994;67:319–333. [Google Scholar]
  10. Chen YH, Edgar JC, Holroyd T, Dammers J, Thönnessen H, Roberts TP, Mathiak K. Neuromagnetic oscillations to emotional faces and prosody. Eur J Neurosci. 2010;31:1818–1827. doi: 10.1111/j.1460-9568.2010.07203.x. [DOI] [PubMed] [Google Scholar]
  11. Davidson RJ, Irwin W. The functional neuroanatomy of emotion and affective style. Trends Cogn Sci. 1999;3:11–21. doi: 10.1016/s1364-6613(98)01265-0. [DOI] [PubMed] [Google Scholar]
  12. de Gelder B, Böcker KB, Tuomainen J, Hensen M, Vroomen J. The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses. Neurosci Lett. 1999;260:133–136. doi: 10.1016/s0304-3940(98)00963-x. [DOI] [PubMed] [Google Scholar]
  13. Dehaene S, Changeux JP, Naccache L, Sackur J, Sergent C. Conscious, preconscious, and subliminal processing: a testable taxonomy. Trends Cogn Sci. 2006;10:204–211. doi: 10.1016/j.tics.2006.03.007. [DOI] [PubMed] [Google Scholar]
  14. Dolan RJ, Morris JS, de Gelder B. Crossmodal binding of fear in voice and face. Proc Natl Acad Sci U S A. 2001;98:10006–10010. doi: 10.1073/pnas.171288598. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Ehrlichman H, Kuhl SB, Zhu J, Warrenburg S. Startle reflex modulation by pleasant and unpleasant odors in a between-subjects design. Psychophysiology. 1997;34:726–729. doi: 10.1111/j.1469-8986.1997.tb02149.x. [DOI] [PubMed] [Google Scholar]
  16. Eimer M, Holmes A. Event-related brain potential correlates of emotional face processing. Neuropsychologia. 2007;45:15–31. doi: 10.1016/j.neuropsychologia.2006.04.022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Ekman P. Emotions revealed: recognizing faces and feelings to improve communication and emotional life. New York: Times Books; 2003. [Google Scholar]
  18. Fenske MJ, Aminoff E, Gronau N, Bar M. Top-down facilitation of visual object recognition: object-based and context-based contributions. Prog Brain Res. 2006;155:3–21. doi: 10.1016/S0079-6123(06)55001-0. [DOI] [PubMed] [Google Scholar]
  19. Gottfried JA, Zald DH. On the scent of human olfactory orbitofrontal cortex: meta-analysis and comparison to non-human primates. Brain Res Rev. 2005;50:287–304. doi: 10.1016/j.brainresrev.2005.08.004. [DOI] [PubMed] [Google Scholar]
  20. Guo J, Guo A. Crossmodal interactions between olfactory and visual learning in Drosophila. Science. 2005;309:307–310. doi: 10.1126/science.1111280. [DOI] [PubMed] [Google Scholar]
  21. Halsband U, Hömberg V. Hemispheric specialization in visual, tactile and crossmodal assembling tasks. Cortex. 1990;26:625–637. doi: 10.1016/s0010-9452(13)80311-5. [DOI] [PubMed] [Google Scholar]
  22. Hannula DE, Simons DJ, Cohen NJ. Imaging implicit perception: promise and pitfalls. Nat Rev Neurosci. 2005;6:247–255. doi: 10.1038/nrn1630. [DOI] [PubMed] [Google Scholar]
  23. Johnson SL, Turner R, Iwata N. BIS/BAS levels and psychiatric disorder: an epidemiological study. J Psychopathol Behav. 2003;25:25–36. [Google Scholar]
  24. Klasen M, Kenworthy CA, Mathiak KA, Kircher TT, Mathiak K. Supramodal representation of emotions. J Neurosci. 2011;31:13635–13643. doi: 10.1523/JNEUROSCI.2833-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Krolak-Salmon P, Hénaff MA, Vighetto A, Bertrand O, Mauguière F. Early amygdala reaction to fear spreading in occipital, temporal, and frontal cortex: a depth electrode ERP study in human. Neuron. 2004;42:665–676. doi: 10.1016/s0896-6273(04)00264-8. [DOI] [PubMed] [Google Scholar]
  26. Krusemark EA, Li W. Do all threats work the same way? Divergent effects of fear and disgust on sensory perception and attention. J Neurosci. 2011;31:3429–3434. doi: 10.1523/JNEUROSCI.4394-10.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Lamme VA, Roelfsema PR. The distinct modes of vision offered by feedforward and recurrent processing. Trends Neurosci. 2000;23:571–579. doi: 10.1016/s0166-2236(00)01657-x. [DOI] [PubMed] [Google Scholar]
  28. Li W, Luxenberg E, Parrish T, Gottfried JA. Learning to smell the roses: experience-dependent neural plasticity in human piriform and orbitofrontal cortices. Neuron. 2006;52:1097–1108. doi: 10.1016/j.neuron.2006.10.026. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Li W, Zinbarg RE, Paller KA. Trait anxiety modulates supraliminal and subliminal threat: brain potential evidence for early and late processing influences. Cogn Affect Behav Neurosci. 2007a;7:25–36. doi: 10.3758/cabn.7.1.25. [DOI] [PubMed] [Google Scholar]
  30. Li W, Moallem I, Paller KA, Gottfried JA. Subliminal smells can guide social preferences. Psychol Sci. 2007b;18:1044–1049. doi: 10.1111/j.1467-9280.2007.02023.x. [DOI] [PubMed] [Google Scholar]
  31. Li W, Zinbarg RE, Boehm SG, Paller KA. Neural and behavioral evidence for affective priming from unconsciously perceived emotional facial expressions and the influence of trait anxiety. J Cogn Neurosci. 2008a;20:95–107. doi: 10.1162/jocn.2008.20006. [DOI] [PubMed] [Google Scholar]
  32. Li W, Howard JD, Parrish TB, Gottfried JA. Aversive learning enhances perceptual and cortical discrimination of indiscriminable odor cues. Science. 2008b;319:1842–1845. doi: 10.1126/science.1152837. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Liddell BJ, Williams LM, Rathjen J, Shevrin H, Gordon E. A temporal dissociation of subliminal versus supraliminal fear perception: an event-related potential study. J Cogn Neurosci. 2004;16:479–486. doi: 10.1162/089892904322926809. [DOI] [PubMed] [Google Scholar]
  34. Livingstone MS, Hubel DH. Do the relative mapping densities of the magno- and parvocellular systems vary with eccentricity? J Neurosci. 1988;8:4334–4339. doi: 10.1523/JNEUROSCI.08-11-04334.1988. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Lundqvist D, Flykt A, Öhman A. The Karolinska Directed Emotional Faces. Stockholm: Karolinska Institute; 1998. [Google Scholar]
  36. Mangun GR, Hillyard SA, Luck SL. Electrocortical substrates of visual selective attention. Cambridge, MA: MIT; 1993. [Google Scholar]
  37. Mathews A, MacLeod C. Cognitive vulnerability to emotional disorders. Annu Rev Clin Psychol. 2005;1:167–195. doi: 10.1146/annurev.clinpsy.1.102803.143916. [DOI] [PubMed] [Google Scholar]
  38. Milner AD, Goodale MA. Two visual systems reviewed. Neuropsychologia. 2008;46:774–785. doi: 10.1016/j.neuropsychologia.2007.10.005. [DOI] [PubMed] [Google Scholar]
  39. Miltner W, Matjak M, Braun C, Diekmann H, Brody S. Emotional qualities of odors and their influence on the startle reflex in humans. Psychophysiology. 1994;31:107–110. doi: 10.1111/j.1469-8986.1994.tb01030.x. [DOI] [PubMed] [Google Scholar]
  40. Modha DS, Singh R. Network architecture of the long-distance pathways in the macaque brain. Proc Natl Acad Sci U S A. 2010;107:13485–13490. doi: 10.1073/pnas.1008054107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Mogg K, Bradley BP, Miles F, Dixon R. Time course of attentional bias for threat scenes: testing the vigilance-avoidance hypothesis. Cogn Emot. 2004;18:689–700. [Google Scholar]
  42. Oya H, Kawasaki H, Howard MA, 3rd, Adolphs R. Electrophysiological responses in the human amygdala discriminate emotion categories of complex visual stimuli. J Neurosci. 2002;22:9502–9512. doi: 10.1523/JNEUROSCI.22-21-09502.2002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Pessoa L, Adolphs R. Emotion processing and the amygdala: from a ‘low road’ to ‘many roads’ of evaluating biological significance. Nat Rev Neurosci. 2010;11:773–783. doi: 10.1038/nrn2920. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Pourtois G, de Gelder B, Bol A, Crommelinck M. Perception of facial expressions and voices and of their combination in the human brain. Cortex. 2005;41:49–59. doi: 10.1016/s0010-9452(08)70177-1. [DOI] [PubMed] [Google Scholar]
  45. Ratcliff R, McKoon G. The diffusion decision model: theory and data for two-choice decision tasks. Neural Comput. 2008;20:873–922. doi: 10.1162/neco.2008.12-06-420. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Sacco T, Sacchetti B. Role of secondary sensory cortices in emotional memory storage and retrieval in rats. Science. 2010;329:649–656. doi: 10.1126/science.1183165. [DOI] [PubMed] [Google Scholar]
  47. Schiffman SS. Physicochemical correlates of olfactory quality. Science. 1974;185:112–117. doi: 10.1126/science.185.4146.112. [DOI] [PubMed] [Google Scholar]
  48. Stein BE, Meredith MA. The merging of the senses. Cambridge: MIT; 1993. [Google Scholar]
  49. Stein BE, Stanford TR. Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci. 2008;9:255–266. doi: 10.1038/nrn2331. [DOI] [PubMed] [Google Scholar]
  50. Turner BH, Mishkin M, Knapp M. Organization of the amygdalopetal projections from modality-specific cortical association areas in the monkey. J Comp Neurol. 1980;191:515–543. doi: 10.1002/cne.901910402. [DOI] [PubMed] [Google Scholar]
  51. Vuilleumier P, Pourtois G. Distributed and interactive brain mechanisms during emotion face perception: evidence from functional neuroimaging. Neuropsychologia. 2007;45:174–194. doi: 10.1016/j.neuropsychologia.2006.06.003. [DOI] [PubMed] [Google Scholar]
  52. Weinberg A, Hajcak G. Electrocortical evidence for vigilance-avoidance in generalized anxiety disorder. Psychophysiology. 2011;48:842–851. doi: 10.1111/j.1469-8986.2010.01149.x. [DOI] [PubMed] [Google Scholar]
  53. Williams LM, Liddell BJ, Rathjen J, Brown KJ, Gray J, Phillips M, Young A, Gordon E. Mapping the time course of nonconscious and conscious perception of fear: an integration of central and peripheral measures. Hum Brain Mapp. 2004;21:64–74. doi: 10.1002/hbm.10154. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Yeshurun Y, Sobel N. An odor is not worth a thousand words: from multidimensional odors to unidimensional odor objects. Annu Rev Psychol. 2010;61:219–241. C1–C5. doi: 10.1146/annurev.psych.60.110707.163639. [DOI] [PubMed] [Google Scholar]
  55. Zelano C, Bensafi M, Porter J, Mainland J, Johnson B, Bremner E, Telles C, Khan R, Sobel N. Attentional modulation in human primary olfactory cortex. Nat Neurosci. 2005;8:114–120. doi: 10.1038/nn1368. [DOI] [PubMed] [Google Scholar]
  56. Zhou W, Chen D. Fear-related chemosignals modulate recognition of fear in ambiguous facial expressions. Psychol Sci. 2009;20:177–183. doi: 10.1111/j.1467-9280.2009.02263.x. [DOI] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES