Disruption in neural phase synchrony is related to identification of inattentional deafness in real‐world setting

Abstract

Individuals often have reduced ability to hear alarms in real world situations (e.g., anesthesia monitoring, flying airplanes) when attention is focused on another task, sometimes with devastating consequences. This phenomenon is called inattentional deafness and usually occurs under critical high workload conditions. It is difficult to simulate the critical nature of these tasks in the laboratory. In this study, dry electroencephalography is used to investigate inattentional deafness in real flight while piloting an airplane. The pilots participating in the experiment responded to audio alarms while experiencing critical high workload situations. It was found that missed relative to detected alarms were marked by reduced stimulus evoked phase synchrony in theta and alpha frequencies (6–14 Hz) from 120 to 230 ms poststimulus onset. Correlation of alarm detection performance with intertrial coherence measures of neural phase synchrony showed different frequency and time ranges for detected and missed alarms. These results are consistent with selective attentional processes actively disrupting oscillatory coherence in sensory networks not involved with the primary task (piloting in this case) under critical high load conditions. This hypothesis is corroborated by analyses of flight parameters showing greater maneuvering associated with difficult phases of flight occurring during missed alarms. Our results suggest modulation of neural oscillation is a general mechanism of attention utilizing enhancement of phase synchrony to sharpen alarm perception during successful divided attention, and disruption of phase synchrony in brain networks when attentional demands of the primary task are great, such as in the case of inattentional deafness.

1. INTRODUCTION

Focusing one's attention on a single task without being distracted by other environmental stimuli is an essential issue for goal‐directed cognition. However, the implementation of such a shielding mechanism might present drawbacks in complex, dynamic, and uncertain real‐life situations. Since the work of Mack and Rock (1998), there is now a strong corpus of evidence that perceptual load might “blind the brain” to an extent that very salient visual stimuli can be neglected (Simons & Chabris, 1999). This inattentional blindness research provides valuable scientific knowledge to understand human performance in everyday life such as car driving or flying aircraft. Unfortunately, these operational situations are rich of accidents whereby humans failed to notice cues in the visual scene or onboard warning system (Gibb & Gray, 2016; Murphy & Greene, 2016). One has to consider that this “inattentional” impairment is not limited to vision and may affect other modalities (Murphy & Dalton, 2016) such as auditory processing (Koreimann, Gula, & Vitouch, 2014).

There has been a growing interest over the last decade to investigate the phenomenon known as inattentional deafness that takes place under demanding visual load settings (Kreitz, Furley, Simons, & Memmert, 2016; Macdonald & Lavie, 2011; Molloy, Griffiths, Chait, & Lavie, 2015). Inattentional deafness (blindness) is closely related to another phenomenon called change deafness (blindness) in that they both involve an absence of awareness of sensory events resulting from a lack of attention (Rensink, 2009, Puschmann et al 2013). They differ in that change deafness (blindness) is related to the lack of perception of a transition of the state of an object or event that is already present, whereas inattentional deafness (blindness) is related to lack of perception of a new object/event. As a matter of fact, the inattentional deafness phenomenon appears to be particularly relevant to account for missed auditory alarm perception that have been reported in the medical domain (Edworthy, 2013) and in aviation (Bliss, 2003; Mumaw, 2017). For instance, experiments conducted in flight simulators, reveal that perfectly audible critical warning alarms could fail to reach awareness (Dehais, Tessier, Christophe, & Reuzeau, 2010; Dehais et al., 2012, 2014). In these contexts, the visual modality is thought to suppress hearing via the implementation of gating mechanisms at the visuo‐auditory integrative stage (Lebib, Papo, de Bode, & Baudonnière, 2003; Molloy et al., 2015) via direct visuo‐auditory connections or through higher level attentional systems (Durantin, Dehais, Gonthier, Terzibas, & Callan, 2017).

In our previous functional magnetic resonance imaging fMRI study (Durantin et al., 2017), the activity of brain regions (inferior frontal gyrus (IFG) and superior medial frontal cortex/presupplementary motor area (pre‐SMA)) associated with the attentional bottleneck (Dux, Ivanoff, Asplund, & Marois, 2006; Szameitat, Vanloo, & Muller, 2016; Tombu et al., 2011) were found to be a neural signature of inattentional deafness. These brain regions were active to a greater extent for poor performance for both auditory alarm detection (participants have to press a button to the presence of audio alarms—misses > hits) and the primary piloting task (participants have to fly through a simulated Red Bull air race course—gates missed > gates passed). It is suggested that when the processing load in the primary task is too high, these attentional bottleneck regions act to preserve selective attention to brain processes involved with the primary visual piloting task while attenuating attentional resources to nonprimary tasks (based on Craik's (1948) hypothesis, concerning the limited processing capacity of the brain). In support of this hypothesis, reduced connectivity was found from the IFG (attentional bottleneck region) to the auditory processing regions of the superior temporal gyrus (STG) during occurrence of inattentional deafness (Durantin et al., 2017). However, unlike the magnetoencephalography (MEG) study conducted by (Molloy et al., 2015), our fMRI study did not reveal reduced activity in auditory brain processing regions as a result of inattentional deafness. Although fMRI has good spatial resolution its temporal resolution of underlying brain activity is somewhat poor compared to other methods such as electroencephalography (EEG) and MEG. It is possible that fMRI may lack the temporal resolution to detect changes in auditory processing given the conditions of the experiment.

Using EEG some studies investigated inattentional deafness in the aviation context with low‐fidelity PC‐based simulators. During the task, a tone was presented, either standard, which participants were instructed to ignore, or deviant (“the alarm”), to which they were to attend, while performing a landing decision task. Scannella, Causse, Chauveau, Pastor, and Dehais (2013) found that auditory N100 event‐related potential (ERP) amplitude was lowered during the difficult flight simulation task. The auditory N100 is associated with neural activity in the auditory cortex (Hall, 1992; Picton et al., 1999; Verkindt, Bertrand, Perrin, Echallier, & Pernier, 1995). Additionally, Giraudet, St‐Louis, Scannella, and Causse (2015) reported auditory P300 amplitude reduction during the difficult flight simulation task. Consistent with these results, additional studies using EEG, MEG, and electrocorticography (ECoG) have also found attention to modulate auditory N100 amplitude (Molloy et al., 2015; Neelon, Williams, & Garell, 2006, 2011; Ponjavic‐Conte, Dowdall, Hambrook, Luczak, & Tata, 2012; Ponjavic‐Conte, Hambrook, Pavlovic, & Tata, 2013) and in some cases auditory P200 amplitude (Neelon et al., 2006) and P300 amplitude (Molloy et al., 2015) as well. In general, attention is thought to increase the magnitude of the peak of the N100 ERP component (Hillyard, Hink, Schwent, & Picton, 1973; Näätänen, Teder, Alho, & Lavikainen, 1992).

While studies investigating evoked potentials are informative they do not provide insight into the spectral and oscillatory/phasic properties thought to be of particular importance for perceptual, motor, and cognitive neural processing (Basar, 1999a, 1999b; Calderone, Lakatos, Butler, & Castellanos, 2014; Klimesch, Sauseng, Doppelmayr, Gruber, & Sauseng, 2004; Nash‐Kille & Sharma, 2014; Palva, Palva, & Kaila, 2005; VanRullen, Busch, Drewes, & Dubois, 2011; Giraud & Poeppel 2012). With regards to perception and attention intertrial coherence (ITC) is of considerable interest. ITC represents the trial‐to‐trial similarity in the frequency specific phasic oscillations of neural activity in relation to stimulus presentation (Makeig, Debener, Onton, & Delorme, 2004). The phase consistency across trials in relation to stimulus onset is likely a result of resetting of the phase of frequency specific intrinsic oscillations (called phase resetting; Hanslmayr et al., 2007b; Makeig et al., 2004, 2002; Sauseng et al., 2007). However, it may also be related to facilitation/disruption of frequency specific intrinsic oscillations (without phase resetting), and induction of new oscillatory activity where there was negligible frequency specific intrinsic oscillation (Busch, Dubois, & Van Rullen, 2009; Hanslmayr et al., 2007b; Sauseng et al., 2007). Facilitation and disruption of oscillation is modulated in part by functional connectivity to brain regions involved with attention (Marshall, O'Shea, Jensen, & Bergmann, 2015). Many studies have implicated ITC (phase synchrony) in attentional and perceptual processing (Busch et al., 2009; Busch & Van Rullen, 2019; Calderone et al., 2014; Hanslmayr et al., 2007a, 2007b; Hanslmayr, Gross, Klimesch, & Shapiro, 2011; Hanslmayr, Volberg, Wimber, Dalal, & Greenlee, 2013; Low & Strauss, 2009; Mathewson, Gratton, Fabiani, Beck, & Ro, 2009; Mathewson et al., 2012; Ponjavic‐Conte et al., 2012, 2013; VanRullen et al., 2011; Yamagishi, Callan, Anderson, & Kawato, 2008).

Within the realm of auditory processing, ITC in the theta and alpha frequency range (4–12 Hz) from ∼80 to 160 ms poststimulus onset is reduced as a result of distraction of selective attentional auditory processing (Ponjavic‐Conte et al., 2012, 2013). It should be pointed out that while there were significant differences in ITC as a result of distraction there were no statistically significant differences in the spectral power (Ponjavic‐Conte et al., 2012). Interestingly, the reduced ITC in the theta frequency range is correlated with N100 ERP mean amplitude and may be responsible for the decrease in N100 peak with greater distraction (Ponjavic‐Conte et al., 2012, 2013). This finding is corroborated by an additional study that also identifies ITC in the theta and low alpha band (6–10 Hz) as contributing to the auditory N100 ERP that is modulated by attention (Low & Strauss, 2009).

Two ways in which phase synchrony of neural oscillations as measured by ITC may be involved with attentional modulation of perception are by (a) increasing the gain of the fixed latency responses in relevant sensory neuronal groups improving perception (Sensory Gain Control Theory of attentional enhancement) (Ponjavic‐Conte et al., 2013). According to this theory, misses are a result of a decrease in facilitation of phase coherence. For the gain‐control theory, one may expect that ITC correlation with overall hit rate and miss rate would be in the same frequency and latency range as it is the degree of gain of the same sensory features that accounts for whether the item is heard or missed. One might expect misses and hits to be processed similarly with the difference being the gain in a specific frequency at certain latency. (b) Selective disruption of the coherency in sensory networks not directly related to the primary task at hand resulting in attenuating perception (disruption of phase coherence theory of selective attention). For the disruption of phase coherence theory, one may expect that misses and hits are processed in a fundamentally different ways as it is thought to be a disruptive modulatory process that accounts for misses as a result of selective attention arising from an attentional bottleneck. Hits, on the other hand, are likely to occur when the attentional bottleneck processes are not in play and therefore this disruption would not be expected to occur.

In the present article, we investigate auditory awareness in flight in a real airplane cockpit with dry wireless EEG. This goal was challenging as only a few studies have considered the use of such devices in real aircrafts, due to the reduced ability to control extraneous variables and the amount of motion‐related or electromagnetic noise in such environments (Callan, Durantin, & Terzibas, 2015; Dehais, Roy, Durantin, Gateau, & Callan, 2017; Scholl et al., 2016). This experiment investigates attentional processing during multitasking, by employing an auditory odd‐ball paradigm in which the pilots respond by button press when hearing a deviant chirp sound. An instructor was present on all flights and in charge of initiating the various scenarios to induce high perceptual and cognitive loads for the pilot. This experimental paradigm was designed to induce high workload piloting situations that are prone to elicit inattentional deafness with a sufficient rate to perform analyses over the misses versus hits contrast. One novel aspect of this study was to focus on the analysis of ITC to characterize the mechanisms at the origin of inattentional deafness under critical real‐life settings that are difficult to simulate in the laboratory. It was predicted that the occurrence of inattentional deafness would be linked to a decrease in neural synchrony identified by ITC to the auditory event, as a result of a reduction of attentional oscillatory modulation.

2. METHODS

2.1. Participants

The participants included in this study consisted of thirteen male pilots and student pilots from 20 to 47 years of age (mean = 25.3, SE = 2.4). Twelve of the participants were right‐handed and one was left‐handed. The participants had the following flight experience: total number of flight hours (mean =100.2, SE = 28.6, range from 26 to 350 h), solo flight hours (mean = 41.3, SE = 21.1, range from 2 to 280 h), and time on DR400 airplane used in experiment (mean = 35.8, SE = 14.1, range from 0 to 170 h). All participants had normal hearing. Normal hearing was based on subjective report. It should be noted that pilots require normal hearing as assessed by medical examination using pure tone audiometry and that all participants in this study were pilots with a current valid medical evaluation. Five separate participants were not included in the study because of machine failures, the EEG data was too noisy, and/or they did not have an adequate number of misses on the audio task. The experiment was approved by the European Aviation Safety Agency (EASA60049235) and supported by the AXA Research Fund (“Neuroergonomics for Flight Safety”). The experiment was carried out in accordance with the principles expressed in the WMA Declaration of Helsinki.

2.2. Inflight experiment

The experiment consisted of an auditory oddball detection task that took place while piloting an airplane under high workload conditions. The two sounds used in the experiment consisted of an auditory chirp from either 2 to 4 kHZ or from 4 to 2 kHz with a duration of 100 ms. These chirp stimuli controlled for frequency and power with the only difference between stimuli being the direction of the sweep. The deviant stimuli serving as the audio alarm to be detected and responded to by button response were presented on 20% of the trials, while the standard sound that did not require any response was presented on 80% of the trials. The participants were instructed to respond quickly by pushing a button attached to the flight control stick (Figure 1) after hearing the deviant alarm stimuli. The two chirp sounds were easily discriminable and audible even with the engine at full throttle in flight. The chirp stimuli used as the deviant alarm (2–4 or 4–2 kHz) were counterbalanced across participants. In total, there were ∼900 stimuli (720 standards and 180 deviants, varying slightly depending on the duration of the flight) presented in random order with an interstimulus interval ranging between 2 and 5 s (randomly determined). Prior to the inflight experiment the participants were given a pretest experiment within the airplane with the engine off to familiarize them with the stimuli and task. The pretest consisted of 100 trials (80 standards and 20 deviants).

Figure 1.

Experimental equipment and setup. Top: DR‐400 Robin 4 seat airplane used in the experiment. Bottom left: Button response unit attached to the control stick. Used by the participant to identify when an audio alarm was heard. Bottom right: The configuration of individuals in the airplane consisted of the participant (pilot) in the front left seat (shown here wearing the Cognionics HD‐72 dry‐wireless 64 channel EEG system), the certified flight instructor in the front right seat, and the research engineer in the back right seat (experimental computer shown on research engineer's lap)

The sound intensity of the stimuli and background environmental noise in the cockpit were measured with a sound level meter. The chirp sounds for the stimuli were presented at ∼85 dBA. The background environmental noise was ∼89 dBA during max thrust (take off), 86 dBA during cruise thrust, and 79 dBA during engine idle in flight (simulated engine out). The reported real‐ear tested attenuation characteristics of the Clarity Aloft headset we used is reported to be 29 dB. The approximate signal to noise ratio depending on the phase of flight ranged from ∼25 to 35 dB.

The piloting task during the audio detection task was intended to place a high workload on the participants. In each flight, there were 3 people in the airplane (DR400 Robin), the participant (pilot) in the front left seat, the certified flight instructor in the front right seat, and the research engineer in the back right seat (Figure 1). Before each flight, the participant was asked to make a flight plan to a specific aerodrome and were told that we would be recording their brain activity by EEG while they piloted the plane and carried out the audio detection task. All flights originated and terminated at the Toulouse Lasbordes Aerodrome. The audio experiment started during takeoff and would end before or during landing. The experiment was approximately one hour in duration. The instructor initiated various scenarios inflight to induce high perceptual and cognitive loads based on the pilot's ability. These scenarios included the following:

1. Navigation to a diverted flight plan. Although the participants were asked to make a flight plan before flight, the instructor would ask them to navigate using charts to a grass airstrip (Aerodrome de Gaillac—Lisle‐sur‐Tarn) that was difficult to see from the air (Figure 2).

   Figure 2.

   

   In‐flight navigation to a grass airstrip. Top: Pilot (shown here wearing the Cognionics HD‐72 dry‐wireless 64 channel EEG system) looking at aeronautical charts to navigate to an airfield with a grass airstrip while simultaneously carrying out the audio task of responding by button press to audio alarm stimuli. Bottom: The grass airstrip (in the center of the image) can be quite difficult to see from the air. One needs to find landmarks on the aeronautical chart (e.g., a river) to help locate the grass airstrip amongst all the other green fields

2. Simulated engine failure and landing at a grass airstrip. The instructor would simulate the engine failure by pulling the throttle to idle. When approaching the Aerodrome de Gaillac, the instructor would simulate the engine failure and the pilot would have to make the approach and landing if deemed safe by instructor on the grass airstrip. The pilot was not to engage the throttle until instructed to do so.

3. Simulated engine failure and off‐field emergency landing procedures. The pilot was to determine a site to land the plane (usually an open field such as a farm) and to make an approach under engine off conditions. The instructor would tell the pilot when to engage the throttle to abort the landing. This usually occurred when it was obvious whether the landing could be made or not given the conditions of the pilot selected landing site (distance to trees or other obstructions such as power lines on approach).

4. Low‐altitude circuit patterns. The participant would engage in a series of touch‐and‐goes (landings and takeoffs) at Aerodrome de Gaillac. The above ground level altitude of the circuit (from 500 to 1000 feet) was determined by the instructor based on ability of the pilot and conditions at the aerodrome. In some cases, pilots were also instructed to make landings without flaps depending on their ability. During the inflight experiment, the instructor took care of all radio communications. The radio communications channel to the participant (pilot) was turned down low to reduce interference with the audio experiment. The audio stimuli were clearly audible even during radio and intercom communication by the instructor.

EEG activity was collected using the Cognionics HD‐72 dry‐wireless 64 channel EEG system (Cognionics, Inc., San Diego) (Figures 1 and 2). The sampling rate was 500 Hz with 24‐bit analog to digital conversion. Active shielding covering all of the electrodes in the headset is used to minimize external noise pickup and artifacts. The Cognionics HD‐72 headset contains three accelerometers (up–down, side–side, and front–back axes) that can be used to detect head movement. For more technical details regarding the Cognionics HD‐72 EEG system used in this experiment, see Callan et al. (2015). The Cognionics HD‐72 EEG system has been verified in previous experiments to be able to pick‐up auditory related brain activity during flight, even in an open cockpit biplane (Callan et al., 2015). In this experiment presented, raw EEG data were collected on a micro SD card located on the headset. Triggers for the onset of the audio stimuli (deviants and standards) as well as button press responses were additionally recorded on the SD card synchronized with the EEG data. The audio experiment was conducted using a computer safely affixed to the airplane and kept in the back seat by the research engineer. The experimental computer was used to present the audio stimuli to the pilot's aviation headset (Clarity Aloft Pro) through the auxiliary input, to collect button press responses via USB, and to collect flight parameters (ILevil2 AW: Attitude Heading Reference System and GPS Navigation). The following flight parameters were recorded: latitude, longitude, altitude, ground speed, vertical speed, roll, pitch, inclination (slip‐indicator in degrees), turn‐rate, and G‐load. All data were synchronized with respect to the EEG using triggers. The triggers were sent wirelessly from the experimental computer to the Cognionics headset using a specialized USB module. For seven of the participants, electrocardiographic and respiration data were also collected, however, was not used in analysis of the experiment presented here.

2.3. EEG processing steps

The processing of the EEG data was conducted using the EEGLAB toolbox (Delorme & Makeig, 2004). For each participant, the following processing steps were conducted:

• The continuous raw data were band passed filtered using a Hamming windowed Sinc FIR filter from 1 to 30 Hz.

• Automatic channel rejection was conducted based on flat channel duration, poor correlation to robust estimate based on other channels, and excessive line noise (default parameter values were used). The mean number of channels remaining out of 64 after channel rejection was 26.4 (SE = 1.7, range = 16–38). The location and number of electrodes rejected were different for every participant. For this reason, a topographic representation of the amplitude distribution and source localization was not presented in this study.

• Automatic subspace reconstruction (Mullen et al., 2013) was used to remove nonstationary high‐variance signals from the EEG by means of interpolation of components that exceed a threshold relative to the covariance of the calibration set of relatively clean data segments (standard deviation cutoff for removal of bursts = 20; Windowed Criterion = 0.25). The results of ASR with and without removing the time windows that were not repaired completely were computed.

• Infomax independent component analysis (ICA) was used over the results of the ASR with the unrepaired time windows removed. The weights of the ICA were then applied to the ASR results without the time windows removed to obtain the full dataset.

The epochs for deviant hits, deviant misses, and standard stimuli (that were not false alarms) 1 s before and 1.5 s after stimulus onset were extracted from the continuous data. Stimuli with button press responses that were >2 s were deemed to slow and counted as misses. The mean number of this type of stimuli across participants with late responses that were counted as misses was 3.46 (SE = 0.79, range = 0–8). The SASICA (Chaumon, Bishop, & Busch, 2015) and the ADJUST (Mognon, Bruzzone, Jovicich, & Buiatti, 2011) EEGLAB toolboxes were used over all of the events to determine artifact based independent components. For 4 of the participants, ADJUST analysis failed, likely due to the distribution of channels remaining after channel rejection. For these participants, artifact components were determined only by the SASICA parameters: autocorrelation, focal components, and signal‐to‐noise ratio. The mean number of artifact components found across participants was 8.3 (SE = 1.07, range = 3–15). A single nonartifact independent component with the greatest projected mean variance showing an auditory ERP over the standard trials was manually selected for each participant for further analyses.

2.4. Event‐related potentials

ERPs were determined for hits and misses for each participant. Two analyses were conducted. One analysis was over the selected independent component activation power. The other analysis was at the electrode level (Cz) of the projected selected independent component. In cases were the Cz electrode was not present (6 of the 13 participants), the values were interpolated using functions within EEGLAB. The trials used for the ERP analyses were baseline normalized using data from −200 to 0 ms prior to stimulus onset. Statistical significance of the difference between Hits and Misses was determined by means of bootstrap statistics using 10,000 random selections with replacement. Bootstrap resampling has advantages over parametric statistical tests (t‐tests, ANOVAs, etc.) in that it does not assume normal distribution and homoscedasticity of the value of interest or the error terms (Efron & Tibshirani, 1994).

2.5. Intertrial coherence

ITC (also referred to as phase‐locking factor) was computed over the trials (hits, misses, and standards) from 1,000 ms before stimulus onset to 1,500 ms after stimulus onset using a Morlet wavelet for time frequency decomposition (default value for cycles = [3 0.5] was used) from 3 to 30 Hz. The common base setting was used to compute and compare the ITC means between the two conditions of interest. The resulting time frequency representation for the ITC was from −444 to 942 ms and from 3 to 30 Hz. The sample sizes of the various conditions (hits vs misses) were made to be equal by random sampling. One thousand random selections of the trials were used for the multiple time frequency analyses upon which the mean value of the ITC was determined for hits, misses, and the comparison between the two conditions. These mean ITC values for the various conditions for each participant were then used for second level random effects analyses between the participants.

The second level random effects analyses were carried out by means of bootstrap statistics using 10,000 random selections with replacement of the mean ITC images of the 13 participants in this study. The stimulus time range was from 8 to 518 ms poststimulus onset, and the baseline time range was from −444 to −20 ms prestimulus onset; the frequency range assessed was from 5.1 to 17.5 Hz. The mean of the baseline time range of the resultant bootstrap analyses was subtracted from each time element of the stimulus time range for all 10,000 images. For each of the elements of the time frequency analysis, the sampling distribution is determined and assessed relative to it being greater than or less than 0. The number of samples falling below 0 out of 10,000 determines the estimated p value for t for that time frequency element. For cases when there were no samples falling below 0, p values of 1/10,001 were given.

The relationship between performance (hit rate and miss rate) and ITC was determined by bootstrap analyses at the random effects level. In this analysis, the correlation between the participants performance measure (miss rate or hit rate depending on the interest of the direction of the correlation) and the elements of the ITC analyses for hits and misses are determined for each of the 10,000 bootstrap iterations. The sampling distribution of the correlation value (r) for the 10,000 iterations was determined and assessed relative to it being greater or less than 0.

To assess the relationship between head movement and ITC, correlation bootstrap analyses were conducted at the random effects level. In this analysis, the correlation between the participants head movement (defined by the (a) mean absolute acceleration within the time window from −1000 to 1500 ms; (b) maximum absolute acceleration within the time window −1,000 to 1,500 ms; (c) mean sum of the absolute difference between samples within the time window −1,000 to 1,500 ms) in the up–down, side–side, and front–back axes during each trial and the elements of the ITC are determined for each of the 10,000 bootstrap iterations. The sampling distribution of the correlation value (r) for the 10,000 iterations was determined and assessed relative to it being greater or less than 0.

The same procedure was used to correct for multiple comparisons for all the analyses. The false discovery rate (FDR) correction (Benjamini & Yekutieli, 2001) was employed across the p values of the multiple tests assessed at p < .05.

3. RESULTS

3.1. Behavioral results

The mean hit rate was 61.9% (SE = 4.5%, range = 34.8%–84.3%) and the mean false alarm rate was 1.2% (SE = 0.29%, range = 0%–3.1%). The mean d′ was 2.73 (SE = 0.066, range = 2.41–3.2). There were ∼180 deviant sound trials that served as the target (20% of trials) and 720 standard trials that served as the distractor (80% of trials). Because of some machine failures there were a couple of trials that were missing for a few subjects. For one of the participants a considerable number of trials were missing, there were only 136 deviant trials and 532 standard trials (Experiment cut short due to battery failure). The subjective workload estimate on a scale from 1 to 7 (1 being normal workload and 7 being overwhelming workload) ranged from 4 to 7 with a median of 5 and a mean of 5.23 (SE = 0.23). A Wilcoxon sign rank test verified that the workload ratings fell above a rating of 4 within the difficult range (p < .001). The mean absolute difference between actual and subjective hit rate was 12.8% (SE = 3.5%, range = 3.3%–40.6%). The correlation between subjective hit rate and actual hit rate was r = 0.57 (p < .05). The mean response time for hits across participants was 792 ms (SE = 24 ms, range = 661–945 ms).

The difference in flight parameters (altitude, vertical speed, roll, pitch, yaw, inclination, turn rate, and G load) at the time of stimulus presentation (mean of data points from 1,000 ms just prior to stimulus onset to 1,500 ms after stimulus onset) was compared between hits and misses. The following flight parameters were found to significantly differ between hits and misses correcting for multiple comparisons (p < .05 two‐tailed corrected): vertical speed, roll, pitch, inclination, and G load (Table 1).

Table 1.

Flight parameters for audio alarm hit and miss conditions

Flight parameter	Hit mean (SE)	Miss mean (SE)	T	p
Altitude feet	1894 (54.2)	1771 (47.3)	2.11	.056
Ground speed knots	81.81 (2.098)	80.97 (1.228)	0.49	.633
Vertical speed feet/minute	86.16 (38.062)	−98.47 (27.592)	4.25	.0011*
Roll degrees	5.74 (0.321)	6.61 (0.284)	−2.59	.024*
Pitch degrees	−1.25 (0.342)	−2.39 (0.255)	3.45	.0048*
Inclination degrees	3.65 (0.293)	3.29 (0.214)	3.50	.0044*
Turn rate degrees/s	−0.272 (0.111)	−0.591 (0.234)	2.16	.051
G‐Load Gs	0.952 (0.002)	0.963 (0.0025)	−4.44	.0008*

Note. Means and standard error SE for the various flight parameters for audio alarm hit and miss conditions. *Statistically significant at p < .05 two‐tailed correcting for multiple comparisons.

The degree of head movement for hits and misses was assessed by use of the three accelerometers mounted in the Cognionics headset. Three separate measures were used for each trial from 1,000 ms prestimulus onset to 1,500 ms poststimulus onset at 2 ms intervals. These included the following: (a) mean absolute acceleration within the time window; (b) maximum absolute acceleration within the time window; (c) mean sum of the absolute difference between samples within the time window. The results are given in Table 2 for the up–down, side–side, and front–back axes.

Table 2.

Head movement accelerometer values for audio alarm hit and miss conditions

Accelerometer parameter	Hit mean (SE)	Miss mean (SE)	T	p
Mean Abs pitch (g)	0.749 (0.016)	0.780 (0.013)	−5.03	.0003*
Mean Abs roll (g)	0.100 (0.010)	0.112 (0.011)	−3.73	.003*
Mean Abs yaw (g)	0.600 (0.020)	0.571 (0.017)	3.94	.002*
Max Abs pitch (g)	0.902 (0.020)	0.954 (0.015)	−6.31	.00004*
Max Abs roll (g)	0.245 (0.017)	0.286 (0.013)	−4.65	.0006*
Max Abs yaw (g)	0.747 (0.020)	0.741 (0.021)	0.59	.57
Mean sum of Abs difference pitch (g)	3.455 (0.138)	3.891 (0.185)	−4.37	.001*
Mean sum of Abs difference roll (g)	3.307 (0.159)	3.788 (0.214)	−5.67	.0002*
Mean sum of Abs difference yaw (g)	4.122 (0.130)	4.252 (0.158)	−2.11	.057

Note. Abbreviations: Abs = absolute; g = g force; SE = standard error; FWE = family wise error.

Means and standard error SE for the various head movement accelerometer parameters for audio alarm hit and miss conditions. *Statistically significant at p < .05 two‐tailed FWE correcting for multiple comparisons.

3.2. EEG results

The random effects results based on bootstrap statistics of the ERP analyses for the independent component activation and the electrode channel level (Cz) are presented in Figure 3. There were no significant differences between hits and misses when correcting for multiple comparisons across the entire time range from 0 to 800 ms. Using 40 ms time regions of interest centered on the N1 and the P2 peaks (as cited in the literature: N1 = 90 ms, Neelon et al., 2006; Ponjavic‐Conte et al., 2012; P2 = 170 ms, Neelon et al., 2006), significant differences (pFDR < .05) in activity were present for the N1 peak (around 90 ms) for both the IC and Cz channel analyses. Even using uncorrected thresholds no significant difference was found for P2. Using an uncorrected threshold (p < .05), a negative peak difference between hits and misses was also found at around 255 ms for both the IC and Cz channel analyses.

Figure 3.

Average event‐related potentials for hits and misses across all participants for (a) independent component IC activations and (b) electrode channel Cz. Bootstrap statistical analyses were conducted. Region of interest (N1 from 70 to 110 ms; T2 from 150 to 190 ms) corrected false discovery rate FDR thresholds for the difference between hits and misses are denoted by magenta asterisks * and uncorrected thresholds are denoted by black asterisks*

The random effects results based on bootstrap statistics (Section 2) for the ITC analyses are given in Figure 4. ITC (Figure 4) was found to be significant (p < .05 one‐tailed corrected) for hits relative to baseline (predominantly 5–13 Hz, peak between 94 and 234 ms), misses relative to baseline (predominantly 5–8 Hz, peak between 122 and 206 ms), and hits‐misses relative to baseline (predominantly 6–14 Hz, peak between 122 and 234 ms). To test the relationship between behavioral performance and ITC bootstrap correlation, analyses were conducted within the region of interest analysis defined by the time–frequency elements found to be significant for the hits–misses ITC analysis given in Figure 4. A significant correlation (r = .71, p < .05 two‐tailed corrected) for hit rate with that of ITC for hits was present at 13 Hz around 50 ms poststimulus onset (Figure 5a). A significant correlation (r = −0.62, p < .05 two‐tailed corrected) for miss rate with that of ITC for misses was present at 8 Hz around 165 ms poststimulus onset (Figure 5b). ITC was only computed for independent component activations. It should be pointed out that as ITC is a measure of phase coherence not influenced by signal amplitude (Delorme & Makeig, 2004) and computed over individual independent components that the projected ITC is identical across all channels even though the projected signal amplitude distribution varies across channels.

Figure 4.

Poststimulus onset intertrial coherence (ITC) for alarm hits and misses. Results for ITC bootstrap analyses. The mean and statistically thresholded (p < .05 one‐tailed corrected) mean relative to baseline are given for alarm hits, misses, and hits‐misses

Figure 5.

Participant level mean correlation between performance and inter‐trial coherence (ITC). Top shows the time frequency elements showing a significant difference between ITC for hits relative to misses from Figure 3. The arrows depict the location of the strongest significant correlation (p < .05 two‐tailed corrected) for the bootstrap correlation analysis for hits and for misses. (a) The fitted linear regression slope (solid blue line) of the peak time frequency correlation (r = .71) between hit rate and ITC for hits located at ∼13 Hz around 50 ms poststimulus onset. (b) The fitted linear regression slope (solid blue line) of the peak time frequency correlation (r = −.62) between miss rate and ITC for misses located at ∼8 Hz around 165 ms poststimulus onset. The blue asterisks are the individual data points for each participant and the dotted blue lines are the confidence bounds of the fitted linear model

To ensure that the reduction in ITC for misses relative to hits was not caused by potential artifacts resulting from head movement random effects bootstrap correlation analyses were conducted over the ITC values for misses using separately the three different head movement acceleration parameters—(a) mean absolute acceleration within the time window; (b) maximum absolute acceleration within the time window; (c) mean sum of the absolute difference between samples within the time window—for up–down, side–side, and front–back axes as regressors. A region of interest analysis was conducted using the time–frequency elements found to be significant for the hits–misses ITC analysis given in Figure 4. It is predicted that if head movement was responsible for the decrease in ITC there should be a negative correlation between the magnitude of the accelerometer parameters and the magnitude of the ITC values. None of these analyses resulted in statistically significant correlation between head movement parameters and ITC (p > .05 one‐tailed corrected).

4. DISCUSSION

This study identifies ITC in the theta and alpha frequency range as a neural signature of inattentional deafness (Figure 4). This was accomplished for the first time in a real‐world setting (in flight while piloting an airplane) in which the critical nature of the workload associated with the task is real rather than just simulated. This finding is of great general importance in that it identifies one potential mechanism by which attention functions in the brain (modulation of neural oscillation) that is relevant in the real world and not just in the isolated conditions of a laboratory.

Oscillatory coherence is thought to enhance neuronal communication both within neuronal groups in localized brain regions and between neuronal groups in different brain regions (Fries, 2005). With regards to attention modulated phase concentrations that occur after stimulus onset as a result of phase resetting and/or induction of new oscillations, they are thought to be involved with facilitating gating of sensory information as well as binding of stimulus features to promote perception (Clayton, Yeung, & Kadosh, 2015; Ponjavic‐Conte et al., 2013; Womelsdorf & Fries, 2007). The increase in phase coherence could work to enhance the relative gain or effectiveness of incoming sensory signals as well as signals communicating across distant cortical regions by reducing temporal jitter (Hillyard & Annllo‐Vento, 1998; Ponjavic‐Conte et al., 2012, 2013; Voloh & Womelsdorf, 2016). In our study, we find a reduction in theta and alpha band (6–14 Hz) ITC for missed auditory alarms relative to ones that were heard from ∼120 to 230 ms after stimulus onset (Figure 4). Although source localization was not possible in this study, as a result of the loss of many channels during the inflight recording, the timing of the evoked response is consistent with networks involving auditory processing regions in the superior temporal gyrus (Hall et al., 1992; Neelon et al., 2006; Verkindt et al., 1995).

Our finding of a decrease in ITC as one potential mechanism responsible for inattentional deafness is consistent with laboratory based studies of auditory distraction (Ponjavic‐Conte et al., 2012, 2013). In their studies, Ponjavic‐Conte et al. (2012, 2013) suggested that distraction may be the result of an increase in the temporal jitter (‘Distraction Decoherence’) that disrupts theta and low alpha band oscillatory coherence in neuronal groups that is a key component for perception. The main task in these studies was to listen for an auditory target within varying degrees of auditory distraction. Our study differs from theirs in many substantial ways. The primary task for the participant in our experiment involves visual, motor, and cognitive processing related to piloting the aircraft, while the secondary task involves identifying the presence of auditory alarms. Our experiment is therefore better identified as a multitasking paradigm, in which the primary task (visual motor control of piloting the airplane) is continuous in nature (varying with the degree of workload depending on the conditions of flight), and the secondary task of identifying the audio alarms is discrete in nature.

Within the context of inattentional deafness, which is the lack of awareness of an auditory alarm, it is maintained that under high load conditions or those requiring focused attention to the primary task, that attentional modulation of secondary tasks is attenuated or perhaps even the networks involved with the secondary tasks are selectively inhibited (Durantin et al., 2017). This is consistent with decreased N1 ERP amplitude found under high workload conditions in response to auditory stimuli (Molloy et al., 2015; Scannella et al., 2013). We show a similar decrease in N1 amplitude for misses relative to hits in this study (Figure 3). The decrease in ITC for missed alarms (Figure 4) found in this study is consistent with two hypotheses: (a) The attentional mechanisms responsible for increases in oscillatory phase concentration in theta and low alpha band frequencies responsible for enhanced stimulus based perception are reduced for missed alarms relative to those that are heard (reduction in facilitation of phase coherence). (b) There is an inhibition/disruption of the mechanisms involved in allowing for oscillatory phase concentration important for perception in networks not related to the primary task (disruption of phase coherence).

Theta and alpha band coherence in sensory cortices is thought to be in part modulated/induced by interaction with frontal attention networks (Clayton et al., 2015). Depending on the focus of attention and workload demands certain task related or unrelated networks to sensory regions may be selectively enhanced or inhibited respectfully. The medial prefrontal cortex (mPFC also encompassing pre‐SMA), identified in an fMRI study as an attentional bottleneck region differentially active during inattentional deafness (Durantin et al., 2017), has been implicated in the processing of attentional control via modulating synchrony of theta and alpha oscillations in sensory and motor regions (Clayton et al., 2015). Increased theta activity in the visual cortex on a spatial attention task has been conjectured to reflect interaction with top–down attention networks (Yamagishi et al., 2008). Our finding of reduced theta activity for audio misses relative to hits (Figure 4) may also reflect to some degree influences from frontal attention networks. The finding of reduced functional connectivity between frontal and temporal auditory processing regions during periods of inattentional deafness, when attentional bottleneck regions are highly active (Durantin et al., 2017), may suggest the selective attenuation of networks not directly involved with the primary task at hand (piloting the airplane in this case) under high load conditions. Consistent with our study, reduction in auditory perception as a result of distraction may be mediated by disruption of phase coherence in theta and low alpha frequency bands (Ponjavic‐Conte et al., 2012, 2013).

In this study, a potential dissociation in the processes that may be involved for alarms that were heard compared to those that were missed can be seen in the respective correlation between overall alarm detection performance and ITC (Figure 5a,b). Participants that had greater miss rates on the alarm detection task had less theta ITC at around 150 ms poststimulus onset for misses (Figure 5b). Whereas, participants that had greater hit rates on the alarm detection task had greater alpha ITC at around 50 ms poststimulus onset for hits (Figure 5a). Rather than seeing the same relationship between ITC for hits and misses with perceptual performance (consistent with a gain‐control theory of attentional enhancement: predicting misses as the result of a decrease in facilitation of phase coherence), we find differences in both the frequency and time of the relationship which is consistent with the disruption of phase coherence theory of selective attention predicting differential processing for misses occurring under high load conditions (as a result of processes related to attentional bottleneck) in the form of attenuation of oscillatory coherency of selective networks not involved with the primary task at hand.

It should be pointed out that contrary to the findings of this study and that of Ponjavic‐Conte et al. (2012, 2013) alpha frequency has often been shown to have the opposite relation with perceptual performance (decreasing in task relevant brain regions and increasing in task irrelevant brain regions with better performance) (Clayton et al., 2015; Fries, Reynolds, Rorie, & Desimone, 2001; Makeig & Inlow, 1993; Womelsdorf & Fries, 2007). The nature of sustained and selective attention and the multitasking demands are considerably different across the various experiments. More research is necessary to determine the role of various oscillatory frequencies in sustained and selective attention under multitask situations that are the norm in real‐world situations.

Evidence that alarms were missed as a result of inattentional deafness during high load conditions is present in the difference in the various flight parameters. The flight parameters that significantly differed between Misses and Hits were associated with greater maneuvering (roll, pitch, G‐load) during descent (vertical speed, pitch) occurring during simulated engine failure (instructor would pull throttle to idle) and landings when piloting workload is at its highest. It is interesting to note that inclination, which is a measure of how coordinated the maneuvering is (avoiding skids and slips), was significantly lower in the miss over the hit condition (Table 1) suggesting that greater attention was focused on piloting.

One of the challenges of conducting research in real‐world environments is the large number of potential artifacts that may confound the results. By collecting data such as head movement and various flight parameters, many of the potential confounds can be addressed and dismissed.

There was significantly greater head movement for misses over hits as determined by the mean, the maximum, and the change in acceleration (of the pitch, roll, and yaw axes of the accelerometer channels on the cognionics headset) during each trial assessed from −1,000 ms prestimulus onset to 1,500 ms poststimulus onset (Table 2). It is possible that this greater head movement may have been related to reduced absolute ITC values for misses relative to hits that was observed in this study (Figure 4). The results of correlation bootstrap analyses did not reveal any significant relationship between the magnitude of the ITC for misses (nor hits–misses) and the various head mounted accelerometer measures. These results do not support the position that head movement was the primary variable responsible for the reduction of ITC for Misses. However, these greater head movements associated with higher miss rate suggest that the pilots were engaged in more demanding flying task such as nominal and off‐nominal landings that require the rapid scanning of the outside world. Consistently with the previous flight parameters analyses, this finding confirms that demanding situations are prone to induce inattentional deafness as previously demonstrated by several studies (Dehais et al., 2014, 2017; Durantin et al., 2017; Giraudet et al., 2015).

Although we did not record the ambient sound in the airplane to ensure that audio alarm misses were not just a product of acoustic masking we can safely rule out this possibility by looking at the significant difference in the flight parameter of vertical speed between the hit and miss conditions (Table 1). The vertical speed indicates whether the plane is climbing (positive values; engine throttle high) or descending (negative values; engine throttle low). Because the missed alarm condition has negative vertical speed and the hit alarm condition has positive vertical speed, it is highly likely there was more acoustic noise generated from the engine for hits rather than misses. Therefore, it is unlikely that alarm misses in our experiment are just a product of acoustic masking from louder environmental noise.

Taken together, these results show that dry electrode EEG systems can be used under real operational settings. The use of such technique was particularly challenging as the electromagnetic environment of the aircraft and the several motion artifacts are known to affect the EEG signal quality. The implementation of our pipeline based on the state of the art of signal processing techniques show its efficiency to measure the neural mechanisms underpinning human performance under complex real‐life situations. It paves the way to the on‐line monitoring of mental states to dynamically adapt pilot‐cockpit interactions for safer operation. This is of importance, as only a single pilot will be used to pilot/supervise the next generation of civilian transportation aircraft. Thus, the onset of critical situations could be more likely to overwhelm this pilot who could not rely anymore on a second pilot to detect alarms.

5. CONCLUSION

These results contribute to a growing number of neuroergonomic‐based studies that maintain that the brain can better be understood by investigating it in real world like settings rather than simplified isolated conditions that only occur in the laboratory (Adamson et al., 2014; Callan et al., 2015, 2012, 2013; Callan, Falcone, Wada, & Parasuraman, 2016a; Callan, Terzibas, Cassel, Sato, & Parasuraman, 2016b; Durantin, Scannella, Gateau, Delorme, & Dehais, 2015; Durantin et al., 2017; Gateau, Durantin, Lancelot, Scannella, & Dehais, 2015; Scholl et al., 2016). This is specifically important when the critical situations under study cannot be easily simulated in the laboratory. Further research needs to be conducted investigating the neural processes underlying multitasking situations and sustained and selective attention. This will allow for a better understanding of how gain and decoherence mechanisms of attentional modulation may be utilized in the brain. The relationship between transitions in ongoing EEG and inattentional deafness will be investigated in subsequent research. The finding of ITC as a neural signature of inattentional deafness can potentially be used as a feature for a brain computer interface that detects occurrence of missed alarms in real time and provides feedback to the pilot through alternate channels and/or implements appropriate countermeasures through the use of neuroadaptive automation. This study demonstrates the feasibility of conducting experiments in real world situations and opens up the possibility for greater use of neuroergonomic approaches in understanding how the brain works.

Callan DE, Gateau T, Durantin G, Gonthier N, Dehais F. Disruption in neural phase synchrony is related to identification of inattentional deafness in real‐world setting. Hum Brain Mapp. 2018;39:2596–2608. 10.1002/hbm.24026