Hearing While Blinking: Multisensory Attentional Blink Revisited

Abstract

It is well established that cognitive system overload is reflected in the attentional blink (AB), the failure to report a second target when it closely follows detection of a first target within a rapid series of stimuli. However, there is intense controversy concerning the effect of first-target detection in one modality on subsequent dynamics of attentional resources in other modalities. Mixed results were found using an audiovisual AB paradigm: depletion of resources in one modality either impaired performance in the other modality or had no effect. Here, we circumvent the need for task switching by measuring an event-related potential, the mismatch negativity, which reflects implicit auditory change detection without requiring task engagement and is present even for background sounds that participants ignore. Surprisingly, we find that during the visual AB, auditory processing is enhanced rather than inhibited, as would be expected by system overload. We suggest that multimodal attentional resources may be freed rather than engaged during the visual AB. Suppression of irrelevant input may require active control by a central executive, which is preoccupied during the visual AB, and/or there may be no reason to suppress other-modal input since the visual system will miss its second target anyway.

Introduction

Despite the human brain's unsurpassed computational capacity, we fail miserably at processing some events when attention is preoccupied with others (Mack and Rock, 1998; Rees et al., 1999; Simons, 2000; Sergent et al., 2005; Del Cul et al., 2007). Numerous studies have asked if the relevant limitation applies to general or to domain- or modality-specific resources, an issue reflecting fundamental brain integrative function.

One fruitful avenue of investigation employs the attentional blink (AB) phenomenon. AB occurs when subjects are asked to report two successive targets (T1, T2) embedded in a rapid serial visual presentation (RSVP) stream of distractor stimuli. Typically, subjects fail to report T2 when it is temporally close to T1 (Raymond et al., 1992; Chun and Potter, 1995; Kranczioch et al., 2007; Slagter et al., 2007). The failure to generate a reportable T2 representation has been attributed to capture of attentional resources by T1 consolidation processes (Chun and Potter, 1995; Shapiro et al., 1997; Marois et al., 2004; Dux et al., 2009). If the deficit is indeed attributable to limited attentional resources, are these resources unimodal or multimodal, and, in particular, does the visual blink affect detection of conspicuous auditory events?

Several studies addressed effects of AB in one modality on attentional resources in another (supplemental section A, supplemental Table S1, available at www.jneurosci.org as supplemental material). Some found cross-modal AB deficits (Arnell and Jolicoeur, 1999; Jolicoeur, 1999; Arnell and Larson, 2002; Soto-Faraco et al., 2002; Arnell and Jenkins, 2004; Arnell, 2006; Van der Burg et al., 2007; Ptito et al., 2008), but most found no audiovisual AB effect (Duncan et al., 1997; Potter et al., 1998; Soto-Faraco and Spence, 2002; Hein et al., 2006; Van der Burg et al., 2007). However, paradigms with T1 in one modality and T2 in another may confound AB and task-switching effects or mitigate AB effects by voluntary preallocation of attention to the T2 modality (Potter et al., 1998; Soto-Faraco et al., 2002). Thus, learning whether and how visual “blinking” affects other modalities may benefit from a novel approach.

Here we address cross-modal AB effects, avoiding divided attention and task switching by measuring the mismatch negativity (MMN) event-related potential (ERP). MMN is elicited automatically when a series of sounds (standards) is interrupted by a different, infrequent sound (deviant). It is elicited typically without requiring response to the sounds, even when subjects strongly attend other stimuli, hence considered preattentive (Näätänen, 1990, 1991; Paavilainen et al., 1993) (for review, see Näätänen et al., 2007; Winkler et al., 2009). Nevertheless, the MMN is not completely independent of attention modulation (Woldorff et al., 1991, 1993, 1998; Näätänen et al., 1993; Kramer et al., 1995; Singhal et al., 2002; Muller-Gass et al., 2005; Yucel et al., 2005; Zhang et al., 2006) (for review of cross-modal attention effects, see Haroush et al., 2009). Accordingly, we found previously that although MMN presence is robust, its amplitude is modulated by moment-to-moment availability of cross-modal attentional resources (Haroush et al., 2009). Hence, MMN amplitude can serve as a measure of cross-modal AB effects, without interference of task switching or divided attention.

Materials and Methods

Participants.

Data from nine participants (age: 19–26; four men) with normal hearing and normal or corrected-to-normal vision were included in the analysis; data of two additional participants were discarded because of excessive eye movement and eye blink artifacts. The experiment conformed to institutional ethics regulations; participants provided informed consent and received course credit or remuneration for participation.

Stimuli and procedure.

Experiments were run in a dimly lit, noise-attenuated, double-walled acoustic booth. Participants sat 100 cm from a 17 inch Viewsonic Graphics Series G75f CRT monitor (1024 × 768 pixels; refresh rate, 100 Hz). A rapid series of 1° visual angle Latin letters, Hebrew letters, and/or numerals was presented. Stimuli were centered in a white square [200 × 200 pixels; red-green-blue (RGB) values, 255–255-255] at the center of an otherwise gray screen (RGB, 128–128-128). Stimuli (RGB, 0–0-0) were presented at 100% contrast for 90 ms each, without interstimulus interval.

A single block consisted of 129 RSVP trials. Each trial contained 10–18 stimuli, including two targets (T1 and T2) (see Fig. 1). T1 appeared with equal probability in positions 5–7; T2 appeared after a lag of another 1, 3, or 7 positions. Trials ended four stimuli after T2 and were followed by a sequence of 20–23 gray “fixation” symbols (*, #, $, or +, also presented for 90 ms each). A similar sequence preceded the first trial.

Figure 1.

Experimental design and visual performance. a, Trial paradigm (see Materials and Methods). b, Detection of T2 when T1 has been correctly reported, as a function of lag. A typical attentional blink is demonstrated: high performance for lag 1, significant decrease at lag 3, and recovery by lag 7. Error bars indicate SEM.

Subjects were tested in two separate experimental sessions, with a different stimulus set for each (to reduce learning effects). One set consisted of Latin capital letter distractors (A–Z) and numeral targets (1–9); the other consisted of Hebrew letter distractors (; total, 22 letters) and nine Latin lowercase letter targets (r, o, c, w, a, x, z, e, n). Subjects typed the identities of T1 and T2 in the presented order during the fixation period on the keyboard number pad (that served also as a “letter pad,” with the nine letter targets represented by numbers in the order above; subjects were pretrained in this keypad use). Subjects were instructed to always report two targets, even if they had to guess. By requiring guessing (precluding a “don't know” option), we ensured that we did not erroneously designate trials as blink trials just because subjects used a conservative criterion, preferring to answer “don't know” when they are not completely sure of target identity.

Each participant performed six blocks of trials per session. All letters, numerals, and T1–T2 lags of one, three, or seven positions were used across trials, in a pseudorandom order; the same target was not used twice successively.

Auditory stimuli consisted of 90 ms (including 5 ms rise and fall times) complex tones with fundamental frequency and first three harmonics with relative amplitudes of 100:75:50:25, presented via a loudspeaker (Peerless 821615) immediately under the monitor. A series consisting of 90% standard tones (fundamental frequency, 600 Hz) and 10% pitch deviants (fundamental, 540 Hz) was presented at ∼70 dB SPL, synchronized with every fourth visual stimulus. Stimulus presentation ensured an equal number of standard and deviant tones together with T2 at lags 3 and 7 (360 in each). Rarely, we presented T2 without a simultaneous sound (36 trials for each lag of one, three, and seven). However, because of the small number of repetitions, ERP responses of these trials were not analyzed. Otherwise, the order of standards and deviants was random except that each deviant was preceded by at least three standards. Participants were instructed to disregard auditory inputs and concentrate on the visual task.

EEG recording and analysis.

The EEG was recorded from 64 scalp sites (Biosemi Active 2; 24 bit at 256 Hz sampling rate; low-pass filtered at 67 Hz to prevent high-frequency aliasing; amplified ×15). Electrodes were embedded in an elastic cap according to the extended 10–20 system. Additional electrodes were placed on the mastoid processes, at the outer canthi of the two eyes [horizontal electro-oculogram (EOG)] and above and below the right eye (vertical EOG). Off-line, data were referenced to the nose and digitally high-pass filtered (Butterworth zero-phase filter, throughout) with a cutoff of 0.5 Hz. Before segmentation, epochs containing blinks, eye movements, excessive muscle activity, or recording artifacts were rejected after a supervised semiautomatic predefined procedure modified to suit individual variation in the EOG artifact amplitude. EEG segments of 500 ms including 100 ms prestimulus onsets time-locked to auditory stimulus were extracted and averaged separately for each participant, electrode, visual lag, and auditory stimulus (standard or deviant).

MMN was computed at each channel by subtracting the average standard response from the average deviant response occurring at the same visual lag (Schröger, 1998).

Note that the MMN results from subtraction of two similar conditions (other than being responses to standard vs deviant tones): blink-present MMNs were computed by subtracting the response to standards from that to deviants, where both occurred simultaneous with missed T2s; blink-absent MMNs were computed by subtracting the response to standards from that to deviants, both simultaneous with identified T2s. Subtracting the response to standard sounds from that to matched-condition deviant sounds removes the contribution of any visual T2 component of the response, as well as the ongoing responses to previous stimuli, which were common to standards and deviants, leaving only the auditory MMN.

Data reduction.

MMN peak latencies were measured at electrode Fz from the difference waveforms of each subject, filtered with a zero-phase Butterworth bandpass of 1–12 Hz (cf. Sinkkonen and Tervaniemi, 2000), within a window of 80–220 ms from tone onset. The corresponding MMN amplitudes were measured in each individual, channel, and condition by averaging 15 data points around the time of the individual MMN peak. For assessing the effect of the attentional blink separately on the nonsubtracted standard and deviant responses, we measured the amplitudes for each subject and condition around the group average MMN response peak.

Steady-state visual evoked potentials (SSVEPs) resulting from the 11 Hz presentation rate of the RSVP (but note the overlap with auditory fourth harmony) were quantified in the 270 ms period just preceding (not including) T1. Data in this time window were detrended (removal of mean and linear trends), and segments were zero padded and transformed using the fast Fourier transform (Matlab version 2009b). SSVEP amplitudes were quantified as the absolute value of the complex Fourier coefficients at the 11 Hz stimulation frequency, and rescaled to decibels. Based on the topographical distribution of SSVEP amplitudes, we chose a cluster of six electrode sites (PO3, PO4, PO7, PO8, O1, O2) where SSVEP amplitudes were maximal and averaged amplitudes across these six electrodes for statistical analysis.

Results

Subjects reported the identities of two numerals in a rapid stream of Latin letters or two Latin letters in a stream of Hebrew letters (using two stimulus sets to counteract learning effects and maintain a constant proportion of blink trials) (Fig. 1a). Comparing results for intertarget lags of one, three, and seven stimulus positions (stimulus onset asynchronies of 90, 270, and 630 ms), a typical visual attentional blink was demonstrated (decreased T2 identification rate given T1 is correctly reported) (Fig. 1b), with performance high at lag 1, decreased at lag 3, and recovered by lag 7 (one-way repeated-measures ANOVA; F_(1,8) = 44.4; p < 0.001) (Fig. 1).

Concurrent with the visual task, sounds were presented in an oddball paradigm (90% standard, 10% pitch deviants), which subjects were instructed to ignore. We observed the expected MMN waveform: a significant negative difference in ERPs evoked by deviant relative to standard tones (Fig. 2a,b; note that despite the seemingly early negativity, no MMN-like topographies were observed in the earlier latencies of 20–60 ms). By design, in most trials a standard or deviant sound coincided with T2 at lag 3 or 7. After correct identification of T1, some visual targets at lag 3 were incorrectly reported (AB present) and some correctly identified (AB absent), so we could compare the MMN elicited with and without visual AB, respectively, under identical stimulation conditions.

Figure 2.

Enhanced auditory MMN during the visual attentional blink. a, Top, MMN difference waveforms (deviant minus standard; electrode Fz; time 0, auditory stimulus onset) at lag 3 for trials with AB present versus absent. Top left inset, Lag 7 AB-absent MMN (light blue) is similar to the waveform for lag 3 with AB absent and not AB present (red and black lines are similar to those in the main diagram). Bandpass is 0.5 to 67 Hz. Bottom, The difference between the AB-absent and AB-present MMN waveforms (dotted green; dark and light shading, 99 and 95% confidence intervals, respectively) is significant around the grand-average MMN peak (green horizontal bar). b, MMN peak distributions. c, Individual MMN peak amplitudes for AB present versus absent. For all subjects, MMN amplitudes are larger with AB present.

MMN response magnitude depended on lag and visual AB response, as shown in Figure 2a (MMN peak mean ± SE: lag 3, AB absent, −1.47 ± 0.31 μv; AB present, −2.72 ± 0.31 μv; lag 7, AB absent, −1.57 ± 0.42 μv; there were too few AB-present trials in lag 7 to include in the analysis). Surprisingly, we found a significantly larger MMN response for lag 3 AB-present trials, when subjects were presumably engrossed in T1 consolidation, than for AB-absent trials (Wilcoxon signed-rank test; p < 0.004, two-tailed; significant also by parametric statistics under assumption of normality F_(1,8)=18.4; p < 0.003). There was no difference between lag 3 and lag 7 AB-absent trials (Wilcoxon signed-rank test; p = 0.73, two-tailed; not significant also by parametric statistics under assumption of normality F_(1,8) < 1). Importantly, the MMN attentional blink effect was observed on a subject-by-subject basis: all participants showed increased MMN amplitude for lag 3 AB-present trials compared to trials where the AB was absent (Fig. 2c). Examining the time course of the difference wave, the difference between AB-present and AB-absent trials is significant specifically during the time around MMN peak and not during earlier phases, or later ERP attention-related components such as the P3a (Fig. 2a, gray confidence-interval lines).

Thus, we find that the auditory MMN signal is enhanced during the visual attentional blink relative to its amplitude when the blink is absent. One explanation of this surprising finding would be to suggest that orienting attention to the auditory modality, triggered or reflected by enhanced MMN response, promotes the visual attentional blink and not vice versa. However, this hypothesis would predict more blink trials when T2 is coincident with a deviant sound than with a standard one, since standard sounds should trigger fewer attention shifts from visual to auditory processing. This was not the case however: a blink was equally probable at lag 3 whether deviant or standard tones were coincident with T2 (40.5 ± 2.93% vs 39.5 ± 3.7%, respectively). Additional evidence comes from the finding that ERP responses to standards and deviants are affected by the visual AB around the MMN peak time (supplemental Fig. S1, available at www.jneurosci.org as supplemental material) in different ways. This was confirmed by a significant interaction between standard/deviant status and AB-present or AB-absent condition (two-way repeated-measures ANOVA; F_(1,8) = 20.6, p < 0.002) found in addition to a main effect of status (F_(1,8) = 41; p < 0.001). Specifically, the deviant ERPs were more negative (or less positive) during AB-present trials than in AB-absent trials (paired t tests, t₍₈₎ = −2.4; p < 0.02), whereas the response to standard sounds changed in the opposite direction, i.e., they tended to be less positive during no-blink compared to blink trials (paired t test, t₍₈₎ = 1.5; p = 0.09). The trend for the standard response was significant during the postpeak phase of the MMN (paired t test, t₍₈₎ = 2.1; p < 0.03; for deviant tones, t₍₈₎ = −1.9; p < 0.04) (supplemental Fig. S1, dark shading, available at www.jneurosci.org as supplemental material). This bidirectional shift in the ERPs for standards and deviants rules out the possibility that the difference between them seen here is attributable to summation of the visual response with the auditory response. Had the latter been the case, the shift should have been in the same direction for all auditory ERPs. Moreover, had the visual AB been brought about through recruitment of attentional resources by the auditory deviants, we would expect the ERP responses to standards to be the same in the presence or absence of an AB. Finally, it is also possible that rather than the blink affecting the processing of auditory stimuli or the other way around, it is ongoing random fluctuations of attention to the visual task that affect both the probability of an AB and the amplitude of the MMN, because when less attention is devoted to the visual task, more is available for auditory processing and vice versa. To examine the level of attention to the visual task, we measured the SSVEPs elicited by the 11 Hz RSVP stream in the last 270 ms before T1 (thus not contaminated with target detection responses) in a set of posterior electrodes showing maximum SSVEPs. Repeated findings show that spatial selective attention to visual stimuli enhances the power of the SSVEP (Morgan et al., 1996), and recent findings extended this to the case of cross-modal audiovisual attention, showing that the power of the SSVEPs at the frequency of the visual stimuli is correlated with the direction of attention toward or away from the visual stream (Saupe et al., 2009). We found no significant difference in the 11 Hz SSVEP power in the pre-T1 period between blink-present and blink-absent trials (t₍₈₎ = 0.52; p = 0.31, one-tail paired t test), indicating that blink occurrence was not correlated with fluctuations in attention to the visual task (supplemental Fig. S2, available at www.jneurosci.org as supplemental material).

Having shown that a blink was equally probable when the simultaneous sound was a standard or a deviant tone, that the ERPs to both standards and deviants are affected by the blink and in opposite directions, and that blink occurrence is not correlated with reduction in attention to the visual signal, we conclude that it is not the presence of an auditory deviant that triggers the blink, not random fluctuations shifting attention between the two modalities that triggers both blink and enhanced MMN, and not summation of the auditory and visual ERPs that enhances the MMN. Rather, we conclude that it must be the visual attentional blink that affects the auditory signal.

Discussion

Previous AB studies found that when the two targets are presented in different modalities, identification of the second target is either impaired (multisensory AB present) or it is not affected by the presence of the first target (no multisensory AB). For example, in an influential study by Duncan et al. (1997) participants reported two targets in a discrimination task. These targets were either both visual, both auditory, or one visual and one auditory. When both targets were presented in the same modality they found a considerable decrease in T2 response accuracy, i.e., an AB. However, when the modality of the second target differed from that of the first, no AB was observed.

In the present study, we used a novel design that obviated the caveats of task switching. Additionally, unlike previous ERP studies of AB, which investigated the processing of T2 targets and found mainly reduction or delay of the P3 ERP component during the blink, presumably related to impaired target consolidation in working memory (Vogel et al., 1998; Arnell 2006; Kranczioch et al. 2007; Ptito et al., 2008), we examined the effect of a visual AB on the processing of information in a task-irrelevant stream of auditory stimuli, rather than on targets. We find first evidence for the counterintuitive result that, rather than “blinking” with the visual system, auditory processing is facilitated during the visual attentional blink. Thus, the auditory system is not only resistant to negative effects of the visual attentional blink, it actually benefits from visual system preoccupation with T1 consolidation. Using a unimodal AB paradigm, Kristjansson and Nakayama (2002) found that T2 is more likely to be detected if it is spatially remote from T1 than when they occur at the same location. This finding was interpreted as showing suppression at the location of T1 rather than enhancement in spatially remote regions. Markedly, our results emphasize that, at least cross-modally, processing of task irrelevant stimuli is in fact enhanced during the blink. Using a dual-task paradigm, we recently obtained behavioral results supporting improved exploitation of auditory working/sensory memory during the visual attentional blink (Haroush and Hochstein, 2010).

How is it that cognitive overload in one modality facilitates processing in a different modality? Previous accounts of AB suggested that reporting visual targets depends on prior working memory “consolidation,” and thus once T1 is perceived, visual attention switches to consolidation to enable T1 report. Failure to report a second target appearing too close to T1 (e.g., at lag 3) results from this concentration of visual resources in consolidation mode (Chun and Potter, 1995; Shapiro et al., 1997; Marois et al., 2004). We suggest that two main mechanisms, which are hard to separate and likely coexist, facilitate auditory processing in this situation. The first relates to the allocation of limited shared multimodal resources. When the visual system is busy consolidating, shared multimodal sensory attentional resources are released to other modalities, including the auditory system, resulting in increased auditory change detection, manifest in the enlarged MMN. This is consistent with EEG data showing that momentary fluctuations of attention to a visual task inversely affect auditory processing (Haroush et al., 2009). Alternatively, change of attention allocation could be caused by random, ongoing fluctuations of attention to the visual modality, not related to T1 processing. This is unlikely to be the source of our finding, however, since we did not find evidence for overall lower attention to the visual task during blink-present over blink-absent trials, using pre-T1 SSVEPs as an index (supplemental Fig. S2, supplemental section C, available at www.jneurosci.org as supplemental material). Recalling that participants were not performing an auditory task in our experiment, we suggest that after T1 detection, sensory attention resources turn from being focused on the visual modality to being spread across all modalities.

The second mechanism that may be responsible for increased MMN relates to top-down executive control. The augmentation of auditory processing likely reflects reduced top-down suppression of irrelevant information processing. Allocation of attention is controlled by central executive mechanisms that both facilitate top-down, task-related processing and inhibit (or deprioritize) (de Fockert et al., 2001) task-irrelevant, data-driven processes (Weissman et al., 2006, 2009). This mechanism may be related to the central executive controlling working memory (Baddeley, 1996). Previous studies have shown that efficient working memory depends on top-down suppression of irrelevant information by frontally based neural mechanisms (Chao and Knight, 1995, 1997). In the present case, when the central executive is available, it restricts allocation of attention to the nonvisual input, including the auditory stream, curtailing the MMN (Haroush et al., 2009). When consolidation of T1 overloads the central executive, not only does the detection of T2 fail, but so does top-down suppression. Thus, there is a reduction in inhibition (i.e., disinhibition) of auditory processing, resulting in an enhanced MMN relative to the ongoing inhibited state. The reduced suppression of auditory processing could also be attributable to strategic, albeit not necessarily conscious, priority setting by executive circuits. Since there is no advantage in inhibiting auditory processing during the visual attentional blink when T2 will likely be missed anyway, disinhibition may be caused by a guided, ecologically beneficial process increasing system efficiency.

Whereas the MMN was enhanced during the blink, the P3a showed a nonsignificant trend toward reduction of amplitude in blink-absent trials. MMN and P3a have been shown to be dissociable (e.g., Rinne et al., 2006), the MMN putatively related to automatic change detection and the P3a to reorienting of attention. A possible explanation for the trend for a reduced P3a during AB is that the reorienting of attention depends on the availability of executive resources that are lacking when T1 is being processed, whereas the MMN benefits from such preoccupation of executive control. This remains conjectural however.

In summary, our findings suggest that when subjects are engrossed in consolidation of a visual target, leading to a visual AB, multimodal attentional resources are freed and mechanisms responsible for implicit detection of auditory deviants are disinhibited. We note that the enhancement in auditory processing is the result and not the cause of the visual attentional blink, nor does it stem from random fluctuations in allocating attention between the modalities.

By circumventing the need for task switching and examining neural activity directly, the present study thus reveals the complexity and flexibility of the attentional system. Even on very short time scales, when one system stalls for processing an imperative stimulus, other systems gain resources and their performance is enhanced.