Effects of spectral smearing of stimuli on the performance of auditory steady-state response-based brain–computer interface

Abstract

There have been few reports that investigated the effects of the degree and pattern of a spectral smearing of stimuli due to deteriorated hearing ability on the performance of auditory brain–computer interface (BCI) systems. In this study, we assumed that such spectral smearing of stimuli may affect the performance of an auditory steady-state response (ASSR)-based BCI system and performed subjective experiments using 10 normal-hearing subjects to verify this assumption. We constructed smearing-reflected stimuli using an 8-channel vocoder with moderate and severe hearing loss setups and, using these stimuli, performed subjective concentration tests with three symmetric and six asymmetric smearing patterns while recording electroencephalogram signals. Then, 56 ratio features were calculated from the recorded signals, and the accuracies of the BCI selections were calculated and compared. Experimental results demonstrated that (1) applying smearing-reflected stimuli decreases the performance of an ASSR-based auditory BCI system, and (2) such negative effects can be reduced by adjusting the feature settings of the BCI algorithm on the basis of results acquired a posteriori. These results imply that by fine-tuning the feature settings of the BCI algorithm according to the degree and pattern of hearing ability deterioration of the recipient, the clinical benefits of a BCI system can be improved.

Electronic supplementary material

The online version of this article (doi:10.1007/s11571-017-9448-y) contains supplementary material, which is available to authorized users.

Introduction

Communication is one of the most important factors for a satisfactory quality of life. Most humans communicate by speaking, writing, and gesture; however, individuals with long-term severe neurological or muscular diseases—e.g., stroke, motor paralysis, and amyotrophic lateral sclerosis—cannot utilize these conventional means of communication. For decades, many studies have been performed to provide alternative means of communication to such disabled patients (Malaia and Newman 2015; Baek et al. 2013). In conventional brain–computer interface (BCI) studies, variations in brain wave patterns that are induced by external stimuli and mental concentration on a specific stimulus such as a steady-state visually evoked potential (SSVEP), auditory steady-state response (ASSR), or P300 are measured, and a specific command among several choice alternatives is selected based on those measurements.

Several factors affect the performance of the BCI systems, and there have been reports that have quantitatively investigated the effects of hardware and software factors of the system itself or the effects of variations in surrounding situations on the system performance. Commonly, the positions of measurement/reference/ground electrodes and the performances of the utilized device and detection algorithm affect the system performance (Lotte et al. 2007). Additionally, in SSVEP BCI, the flickering frequency, intensity, color, pattern, background, and size of visual stimuli and the arrangement of light sources (e.g., light emitting diodes or patterns on a monitor) can affect the system performance (Allison et al. 2008; Jukiewicz and Cysewska-Sobusiak 2016; Duszyk et al. 2014); in ASSR BCI, the frequency, intensity, and pattern of auditory stimuli and the arrangement of audio sources (e.g., speakers, earphones, or vibrators) can affect the system performance (Matsumoto et al. 2012; Nakamura et al. 2013); and in P300 BCI, the background noise, presentation paradigm, stimuli types (e.g., sound or visual), and user’s attention can affect the system performance (Zhou et al. 2016; Huang et al. 2016; Jin et al. 2015, 2017). However, considering that the values of features that are the basis of BCI selection—e.g., the amplitudes of SSVEP, ASSR, or P300 signals—are calculated from the brain responses that are induced by certain intracranial processes that have not yet been clearly identified, it can be assumed that the pathophysiological statuses of distal stimulus-sensing organs, such as eyes or ears, and those of the sensing and processing parts of the brainstem and brain (Ortner et al. 2011), as well as psychological factors, such as concentration on the stimulus and comfort in concentration (Voicikas et al. 2016), may also affect the values of BCI-related features and, as a result, affect the performance of the BCI system. For example, when a recipient of an auditory BCI system, whether a severely motor-disabled patient or a healthy person, has additional symptoms of sensorineural hearing impairment, such as deterioration of spectral resolution due to the widened auditory filter bandwidth in the inner ear (Bhatt et al. 2001; Op de Beeck et al. 2011), a spectral smearing to the auditory stimulus can be produced that deteriorates the recognized sharpness of the stimulus, which may affect the degree of and comfort in concentration and may also affect the values of the internal parameters of the BCI algorithm. However, as far as we know, there have been few reports that investigated the effects of the degree and pattern of such spectral smearing on the performance of auditory BCI systems.

In this study, we assumed that the spectral smearing of stimuli due to sensorineural hearing impairment affects the performance of an ASSR-based BCI system. To verify this assumption, we constructed smearing-reflected stimuli using an 8-channel vocoder and performed subjective experiments using 10 normal-hearing subjects.

Materials and methods

Constructing smearing-reflected stimuli

To construct smearing-reflected stimuli, an 8-channel vocoder was implemented based on Mulder et al. (2015): (1) an input signal was applied to a Hanning window (window size = 20 ms, 50% overlapping); (2) a short-time Fourier transform (STFT; frequency resolution = 1 Hz) was applied to the output of the Hanning window; and (3) the amplitude components of the STFT output were applied to a rounded exponential (ROEX) auditory filter whose weight is defined by Eq. 1:

$$\text{W}\left(g\right)=\left(1+pg\right)exp\left(-pg\right)$$ 1

where $g$ represents the deviation from the center frequency (f _C) and p represents the sharpness of the filter. The values of p and $g$ were defined as in Eqs. 2–4:

$$g=\left|f-f_C\right|/f_C$$ 2

$$p=4f_C/ERB$$ 3

$$ERB=24.7\mathit{\alpha }\left(0.00437f_C+1\right)$$ 4

where ERB represents the equivalent rectangular bandwidth of an auditory filter as a function of f _C, and $\mathit{\alpha }$ represents a scaling constant that determines the degree of spectral smearing (Mulder et al. 2015; Hansen and Dahl 2014). Then, the output of $\text{W}\left(\text{g}\right)$ and the phase components of the STFT output were inverse transformed to construct smearing-reflected sounds. Next, the constructed sounds were twice applied to a second-order Butterworth infinite impulse response notch filter to eliminate the 6000 Hz component generated by the difference between the sampling rate of the input sound (48,000 Hz) and the upper frequency limit of the vocoder (6000 Hz), which constructs the output of the vocoder. In this study, we assumed three levels of auditory filter bandwidth on the basis of the previous reports of Glasberg and Moore (1986) and Baer and Moore (1994): (1) no smearing (NL): auditory filter bandwidth equivalent to normal hearing (no vocoder); (2) mid-level smearing (ML): auditory filter bandwidth equivalent to moderate hearing loss ($\mathit{\alpha }$ = 3 in Eq. 4), and (3) high-level smearing (HL): auditory filter bandwidth equivalent to severe hearing loss ($\mathit{\alpha }$ = 6 in Eq. 4). Table 1 and Fig. 1 demonstrate the frequency characteristics of the ROEX filters in the vocoder.

Table 1.

Cut-off and center frequencies of the implemented 8-channel vocoder

CH	f C (Hz)	f L (Hz)	f U (Hz)	ERB (ML/HL)
1	141	80	202	120/240
2	293	202	384	169/338
3	521	384	657	243/485
4	861	657	1065	353/706
5	1370	1065	1675	518/1036
6	2132	1675	2589	764/1529
7	3272	2589	3955	1134/2267
8	4978	3955	6000	1686/3372

f _C center frequency, f _L lower cut-off frequency, f _U upper cut-off frequency, ERB equivalent rectangular bandwidth of the auditory filter

Fig. 1.

Scheme and characteristics of the implemented 8-channel vocoder. a Block diagram; b ROEX filter characteristics in the ML and HL setup; c output of the vocoder for S₅₀₀{37} in the ML and HL setup; d output of the vocoder for S₂₀₀₀{43} in the ML and HL setup

In this study, we determined the frequencies of stimuli (37 and 43 Hz modulation frequency), points of measurement (Cz, Oz, T7, and T8), kinds of features (features in “Calculating features and BCI accuracy” section), and method of BCI selection (tenfold cross validation) by referring to a previous report of Kim et al. (2011). Two sinusoidal amplitude-modulated stimuli were synthesized with 100% modulation depth: S₅₀₀{37} and S₂₀₀₀{43}, where 500 or 2000 represents the carrier frequency (CF) in hertz, and 37 or 43 represents the modulation frequency (MF) in hertz. By applying S₅₀₀{37} and S₂₀₀₀{43} to the implemented vocoder, four smearing-reflected stimuli were constructed: S₅₀₀{37}_ML, S₅₀₀{37}_HL, S₂₀₀₀{43}_ML, and S₂₀₀₀{43}_HL. The original S₅₀₀{37} and S₂₀₀₀{43} were used as S₅₀₀{37}_NL and S₂₀₀₀{43}_NL, respectively (Fig. 2).

Fig. 2.

Waveforms of stimuli with no, mid-level, and high-level spectral smearing. a S₅₀₀{37}_NL; b S₂₀₀₀{43}_NL; c S₅₀₀{37}_ML; d S₂₀₀₀{43}_ML; e S₅₀₀{37}_HL; f S₂₀₀₀{43}_HL

Recording setups

The 10 subjects recruited for in this study had no history of neurological diseases and were determined to have normal hearing by a pure-tone audiometry test (seven males and three females, age range: 23–29 years, median age = 26 years, average age = 26 years). Our experimental protocol was approved by the Institutional Review Board of Hanyang University (IRB number: HYI-15-011-3), and written agreements were acquired from each subject. Before the main experiments, each subject sat on a chair in the center of an electrically isolated cage (200 × 120 × 190 cm), was attached to four dry electrodes (g.SAHARA electrode, 7 mm with g.SAHARAclip; g.tec Medical Engineering GmbH, Schiedlberg, Austria) at the C_Z, O_Z, T7, and T8 positions for measurement (Kim et al. 2011) and two hydraulic electrodes at the left/right mastoids (adhesive mastoid electrodes with g.SAHARAclipREF/g.SAHARAclipGND; g.tec) for reference and ground, and was fitted with a headset (MM50 iP; Sennheiser Electronic GmbH, Wedemark, Germany). Then, the electrodes were connected to a commercial EEG measurement device (MOBIlab + with Sahara box; g.tec) that transmitted measurement data to a recording computer via Bluetooth (256 Hz sampling rate and 500 μV sensitivity). After the hardware setup, to verify the relevance of the setup, S₅₀₀{37}_NL was played via the headset with a sound pressure level (SPL) of 65 dB, the subject was asked to concentrate on the stimulus for 13 s, and the signal-to-noise ratio (SNR) between the spectral power at MF and the average spectral power around the MF (±5 bins from the MF) was calculated: the recording setups were repeatedly adjusted until the SNR became ≥2.

Protocols of the subjective tests

Under real-world conditions, some individuals have hearing impairment with symmetric left/right smearing, whereas others have hearing impairment with asymmetric left/right smearing (Kim et al. 2015). Therefore, in the main experiments, we performed two separate experiments.

In the symmetric test, (1) five training trials were performed to familiarize subjects with test protocols, (2) a beep sound with 0.2 s duration was presented to the subject with random direction to indicate the focusing direction for following stimuli (Kim et al. 2011), (3) one of three stimuli sets—[S₅₀₀{37}_NL, S₂₀₀₀{43}_NL], [S₅₀₀{37}_ML, S₂₀₀₀{43}_ML], or [S₅₀₀{37}_HL, S₂₀₀₀{43}_HL]—was presented to both ears for 13 s, and the subject was asked to concentrate on a stimulus sound at beep-indicated direction; and (4) steps 2 and 3 were performed 60 times (3 stimuli sets × 2 concentration directions × 10 repetitions; order of conditions was randomized [Supplement_1]). The procedure was denoted as the SYM test.

In the asymmetric test, (1) five training trials were performed, (2) a beep sound with 0.2 s duration was presented to the subject with random direction to indicate the focusing direction for following stimuli, (3) one of six stimuli sets—[S₅₀₀{37}_NL, S₂₀₀₀{43}_ML], [S₅₀₀{37}_NL, S₂₀₀₀{43}_HL], [S₅₀₀{37}_ML, S₂₀₀₀{43}_NL], [S₅₀₀{37}_ML, S₂₀₀₀{43}_HL], [S₅₀₀{37}_HL, S₂₀₀₀{43}_NL], or [S₅₀₀{37}_HL, S₂₀₀₀{43}_ML]—was heard to both ears for 13 s, and the subject was asked to concentrate on a stimulus sound at beep-indicated direction; and (4) steps 2 and 3 was performed 120 times (6 stimuli sets × 2 concentration directions × 10 repetitions; order of conditions was randomized [Supplement_2]). This procedure was denoted as the ASYM test.

In both tests, each trial was performed using 13 s of concentration then 5 s of rest, and the volume of the stimuli was fixed at 65 dB SPL. To minimize the negative effects of physical and psychological fatigue on the data, each subject visited the testing room twice: one day for the SYM test (about 20 min) and another day for the ASYM test (about 40 min).

Calculating features and BCI accuracy

After performing the main tests, 64 features were calculated as follows (Kim et al. 2011): (1) 8 measurements at the Cz, Oz, T7, and T8 positions (denoted by Cz{37}, Oz{37}, T7{37}, T8{37}, Cz{43}, Oz{43}, T7{43}, and T8{43}); (2) 8 ratios between measurements whose positions were identical and MFs were different (e.g., Cz{37}/Cz{43} or T7{37}/T7{43}); (3) 24 ratios between all possible pairs of measurements with the same MF (e.g., Cz{37}/T7{37} or Oz{43}/T8{43}); and (4) 24 ratios between all possible pairs of measurements with different MFs (e.g., Cz{37}/Oz{43} or T7{43}/T8{37}). For the overall features utilized, measurements were determined as the average spectral densities in the MF ±1 Hz area. Then, for each test condition, three of the 56 ratio features whose differences between left-concentrating and right-concentrating measurements were highest were extracted and determined as primary features for each condition; then, the accuracies of the BCI selections were calculated using those primary features with Euclidean distance tenfold cross validation and confusion matrix analyses. More specifically, for each of the 9 stimuli sets (3 for symmetric and 6 for asymmetric), (1) measurements of the three primary features were converted to a point in the virtual 3-dimensional coordinate whose axis are identical to the 3 primary features; (2) 18 points from 9 left concentration and 9 right concentration were set to reference set, and 2 points from the remaining 1 left concentration (LC__TEST) and 1 right concentration (RC__TEST) were set to test set; (3) average points of the 9 left-concentration points (LC__REF) and the 9 right-concentration points (RC__REF) were calculated, respectively; (4) 2 Euclidean distances between (LC__REF and LC__TEST) and (RC__REF and LC__TEST) were calculated, and the direction whose Euclidean distance is less than the other was determined as the user’s selection; (5) 2 Euclidean distances between (LC__REF and RC__TEST) and (RC__REF and RC__TEST) were calculated, and the direction whose Euclidean distance is less than the other was determined as the user’s selection; and (6) steps 2–5 were repeated 10 times according to the tenfold cross validation protocol.

Results

Table 2 represents the calculated primary features for each smearing condition. In almost all of the conditions and subjects, the kinds of extracted primary features were different from those in the no-smearing condition, which implies that the spectral smearing can affect the optimal settings of the BCI system. Specifically, in two of the 10 subjects (4 and 5), there was no overlap of primary features between the no-smearing and smearing conditions. In addition, features in the last row include the three features for each smearing/no-smearing condition occurred most frequently in the 10 rows above (when multiple features had identical frequencies of occurrence, the features that showed the highest average BCI accuracies during the cross validation when included in the primary features were selected).

Table 2.

The calculated primary features for each test conditions

Subject	SYM test			ASYM test
	NH/NH	ML/ML	HL/HL	NL/ML	NL/HL	ML/NL	ML/HL	HL/NL	HL/ML
1	Oz{37}/Cz{37} Oz{37}/Cz{43} T7{37}/Cz{43}	Oz{37}/Cz{43} Oz{43}/Cz{43} T7{37}/Cz{43}	T7{43}/Cz{43} T7{37}/T8{43} T7{43}/T8{43}	T8{37}/T7{37} T8{37}/T7{43} T8{43}/T7{43}	T8{37}/T7{37} T8{37}/T7{43} T8{43}/T7{43}	Oz{37}/T8{43} Oz{43}/T8{43} T7{37}/T8{43}	Cz{37}/Oz{43} Cz{37}/T7{43} Cz{37}/T8{43}	Cz{37}/T8{37} Cz{37}/T8{43} Oz{37}/T8{43}	T7{43}/Oz{37} T8{43}/Oz{37} T8{43}/Oz{43}
2	Oz{43}/Cz{43} T7{37}/Cz{43} Oz{43}/Oz{37}	Cz{37}/T7{43} Oz{43}/T7{43} T8{37}/T7{43}	T7{37}/Cz{37} T7{37}/Cz{43} T7{37}/Oz{43}	Cz{37}/T7{37} T8{37}/T7{37} T8{43}/T7{37}	Oz{37}/Cz{43} T7{37}/Cz{43} T7{43}/Cz{43}	T7{37}/Cz{37} T7{37}/Cz{43} T7{37}/T8{37}	T7{43}/Cz{37} T7{43}/Oz{37} T7{43}/T8{43}	Cz{37}/Oz{37} Cz{37}/Oz{43} T7{37}/Oz{43}	T7{37}/Cz{37} T7{37}/T8{37} T7{37}/T8{43}
3	Oz{37}/T8{37} Oz{37}/T8{43} Oz{43}/T8{43}	Oz{37}/T7{37} Oz{37}/T7{43} Oz{37}/T8{43}	Oz{37}/T8{43} Oz{43}/T8{43} T7{37}/T8{43}	T7{37}/T8{37} T7{43}/T8{37} T7{43}/T8{43}	Oz{37}/T7{37} Oz{43}/T7{37} Oz{43}/T8{37}	Oz{37}/T7{37} Oz{37}/T7{43} Oz{37}/T8{43}	Cz{37}/T7{43} Cz{37}/T8{43} Oz{37}/T8{43}	Oz{37}/T7{37} Oz{37}/T7{43} Oz{43}/T7{37}	T7{37}/Cz{43} Oz{37}/T8{43} T7{37}/T8{43}
4	Cz{37}/T7{37} Cz{43}/T7{37} T8{37}/T7{37}	Cz{37}/Oz{43} Cz{43}/Oz{43} T7{43}/Oz{43}	T7{37}/Oz{37} T7{37}/Oz{43} T7{43}/Oz{43}	Cz{37}/T7{43} Oz{37}/T7{43} T8{37}/T7{43}	T7{37}/Oz{43} T7{37}/T7{43} T7{37}/T8{43}	T7{37}/Oz{43} T7{37}/T7{43} Oz{37}/T7{43}	Oz{37}/Oz{43} T7{37}/Oz{43} Oz{37}/T8{43}	Cz{37}/T7{43} Oz{37}/T7{43} T8{37}/T7{43}	Cz{43}/T7{43} Oz{37}/T7{43} Oz{43}/T7{43}
5	T8{37}/Oz{37} T8{37}/T7{37} T8{43}/T7{37}	Oz{43}/Oz{37} T8{43}/Cz{37} T8{43}/Oz{37}	T7{37}/Oz{37} T7{43}/Oz{37} T8{43}/Oz{37}	Cz{37}/Oz{37} Cz{37}/Oz{43} T8{37}/Oz{43}	T7{43}/Cz{43} T7{43}/T8{37} T7{43}/T8{43}	Oz{43}/Cz{43} Oz{43}/T7{37} Oz{43}/T7{43}	T8{37}/T7{37} T8{37}/T7{43} T8{43}/T7{43}	Oz{43}/T7{37} Oz{37}/T8{43} Oz{43}/T8{43}	T8{37}/Cz{43} T8{43}/Cz{43} T8{43}/T7{43}
6	T8{37}/T7{37} T8{37}/T7{43} T8{43}/T7{43}	Oz{43}/Cz{37} Oz{37}/T7{37} Oz{43}/T7{37}	Cz{37}/T7{37} Cz{37}/T7{43} Oz{37}/T7{43}	Oz{37}/T7{37} Oz{37}/T7{43} Oz{43}/T7{43}	Oz{43}/Cz{37} Oz{37}/T7{43} Oz{43}/T7{43}	Oz{37}/T7{43} T8{37}/T7{37} T8{37}/T7{43}	Oz{37}/T7{43} Oz{43}/T7{43} T8{37}/T7{43}	Oz{43}/Cz{37} Oz{43}/Cz{43} Oz{43}/T7{43}	Oz{37}/Cz{37} Oz{37}/T7{37} Oz{37}/T8{43}
7	Oz{37}/Cz{37} Oz{37}/T7{37} Oz{37}/T7{43}	Oz{37}/Cz{43} Oz{43}/Cz{43} Oz{43}/T8{43}	T7{43}/Cz{37} T7{43}/T8{37} T7{43}/T8{43}	Cz{43}/T7{43} Oz{43}/T7{43} T8{43}/T7{43}	Cz{43}/Oz{43} T7{43}/Oz{37} T7{43}/Oz{43}	Cz{43}/T8{37} Oz{37}/T8{37} Oz{43}/T8{37}	Cz{37}/Oz{37} T7{43}/Oz{37} T8{37}/Oz{37}	Oz{37}/T7{37} Oz{43}/T7{37} Oz{43}/T8{37}	T7{43}/Oz{37} T7{43}/Oz{43} T7{43}/T8{37}
8	T7{37}/T8{37} T7{37}/T8{43} T7{43}/T8{43}	Oz{37}/Cz{43} Oz{37}/T7{37} Oz{37}/T7{43}	Oz{43}/T7{37} Oz{43}/T7{43} T8{37}/T7{43}	Oz{37}/T7{43} T8{37}/T7{43} T8{43}/T7{43}	T7{37}/Cz{43} T7{37}/Oz{43} T7{37}/T8{43}	T7{37}/T8{37} T7{37}/T8{43} T7{43}/T8{37}	Oz{37}/T7{37} Oz{37}/T7{43} Oz{43}/T7{37}	T7{37}/Cz{43} T7{37}/Oz{43} T7{37}/T8{43}	Oz{37}/T7{37} Oz{37}/T7{43} Oz{43}/T7{43}
9	T7{37}/Oz{37} T7{43}/Oz{37} T7{43}/Oz{43}	Oz{37}/T8{37} Oz{43}/T8{37} T7{43}/T8{37}	T7{43}/Cz{37} Oz{43}/T8{37} T7{43}/T8{37}	Cz{43}/Oz{37} Cz{43}/Oz{43} T7{43}/Oz{37}	Oz{43}/Cz{37} Oz{37}/T8{37} Oz{43}/T8{37}	Cz{43}/Oz{37} T7{43}/Oz{37} T8{43}/Oz{37}	Cz{43}/Oz{37} T7{43}/Oz{37} T8{43}/Oz{37}	Cz{43}/T8{37} Oz{43}/T8{37} T8{43}/T8{37}	T7{43}/Oz{37} T8{43}/Oz{37} T8{43}/Oz{43}
10	Oz{37}/Cz{37} Oz{43}/Cz{37} T7{37}/Cz{37}	T8{37}/Oz{37} T8{43}/Oz{37} T8{43}/T7{37}	Cz{37}/T7{37} Cz{43}/T7{37} Cz{43}/T8{37}	T8{37}/T7{37} T8{37}/T7{43} T8{43}/T7{43}	T7{43}/Cz{37} Oz{43}/T8{37} T7{43}/T8{37}	T7{37}/Cz{37} T8{43}/Cz{37} T7{37}/Oz{37}	T7{37}/Oz{37} T7{37}/Oz{43} T7{43}/Oz{37}	T7{37}/Oz{37} T7{43}/Oz{37} T8{43}/Oz{37}	T7{37}/Cz{37} T7{43}/Cz{37} T7{43}/Cz{43}
Recommended	Oz{37}/Cz{37} T8{37}/T7{37} T7{37}/Cz{43}	Oz{37}/Cz{43} Oz{37}/T7{37} T8{43}/Oz{37}	Cz{37}/T7{37} T7{43}/Cz{37} T7{37}/Oz{43}	T8{37}/T7{43} T8{43}/T7{43} T8{37}/T7{37}	Oz{43}/T8{37} T7{37}/Cz{43} T7{37}/Oz{43}	Oz{37}/T7{43} T7{37}/T8{43} Oz{37}/T8{43}	T7{43}/Oz{37} Cz{37}/T7{43} Cz{37}/T8{43}	Oz{43}/T7{37} Oz{37}/T7{37} Oz{37}/T7{43}	T7{43}/Oz{37} Oz{37}/T7{43} T7{37}/T8{43}

Bold represents primary features that overlapped with the no-smearing (NH/NH) condition

Table 3 presents the results of the cross validation for calculating BCI accuracy for three feature settings: (1) fixed-feature: the kinds of primary features were fixed to the selection in the no-smearing condition regardless of the variations in smearing conditions, (2) adjusted-feature: the kinds of primary features were adjusted in accordance with Table 2, and (3) recommended-feature: the kinds of primary features were adjusted in accordance with the last row of Table 2. The ranges of average accuracy were 45.0–69.0% (56.3 ± 7.8% in mean ± standard deviation format) for the left-concentration and 63.0–71.0% (66.7 ± 3.1%) for the right-concentration when adjusted-feature case. In contrast, the accuracies were 40.0–69.0% (53.8 ± 10.8%) for the left-concentration and 48–71.0% (54.0 ± 7.5%) for the right-concentration when fixed-feature case, which demonstrates the efficacy of the primary feature optimization according to the smearing condition for maintaining a high and consistent level of BCI accuracy. In addition, the accuracies in the right-concentration were higher than those in the left-concentration for all conditions in the SYM test; that is, they were 16, 11, and 13% for the no-smearing, mid-smearing, and high-smearing conditions, respectively, in the adjusted-feature cases and were 16, 6, and 5% for the same conditions in the fixed-feature cases on average. Furthermore, when the recommended features in Table 2 were applied to all subjects identically, the resultant accuracies fell between the fixed-feature and the adjusted-feature cases for most smearing conditions, which implies that the use of the recommended features can also improve the stability of the system compared to the use of the conventional (fixed-feature) systems.

Table 3.

The results of Euclidean distance tenfold cross validation for calculating BCI accuracy or various smearing conditions (mean ± standard deviation format)

Primary features	Concentration direction	SYM test			ASYM test
		NH/NH	ML/ML	HL/HL	NL/ML	NL/HL	ML/NL	ML/HL	HL/NL	HL/ML
Adjusted	L/R	55.0 ± 18.4/71.0 ± 14.5	55.0 ± 15.1/66.0 ± 19.0	50.0 ± 22.6/63.0 ± 22.1	65.0 ± 12.7/69.0 ± 9.9	45.0 ± 19.6/65.0 ± 19.6	52.0 ± 12.3/64.0 ± 17.1	69.0 ± 12.9/63.0 ± 9.5	63.0 ± 11.6/70.0 ± 9.4	53.0 ± 16.4/69.0 ± 15.2
Fixed	L/R	55.0 ± 18.4/71.0 ± 14.5	42.0 ± 19.3/48.0 ± 27.0	45.0 ± 16.5/50.0 ± 20.0	69.0 ± 13.7/48.0 ± 24.4	40.0 ± 20.0/51.0 ± 28.8	48.0 ± 22.0/61.0 ± 12.0	69.0 ± 19.1/50.0 ± 18.3	59.0 ± 18.5/55.0 ± 15.8	57.0 ± 11.6/52.0 ± 7.9
Recommended	L/R	56.0 ± 20.1/60.0 ± 15.6	59.0 ± 13.7/51.0 ± 17.9	54.0 ± 19.6/57.0 ± 19.5	65.0 ± 15.8/60.0 ± 18.3	41.0 ± 26.1/59.0 ± 15.2	56.0 ± 17.8/55.0 ± 19.0	68.0 ± 14.0/63.0 ± 9.5	72.0 ± 9.2/66.0 ± 11.7	64.0 ± 16.5/56.0 ± 17.1

Unit = %

L left-concentration, R right-concentration

Table 4 demonstrates the results of the confusion matrix analysis in the aspect of true-positive, true-negative, false-positive, and false-negative (in frequency). Effect of feature adjustment was the most remarkable when the instructed direction was right (lowest failure and highest success).

Table 4.

Frequencies for true-positive (instructed L and selected L), true-negative (instructed L and selected R), false-positive (instructed R and selected L), and false-negative (instructed R and selected R) for 3 symmetric and 6 asymmetric conditions

Order of test conditions: NH/NH; ML/ML; HL/HL; NL/ML; NL/HL; ML/NL; ML/HL; HL/NL; HL/ML		Instructed direction (F/A/R)
		L	R
Selected direction (F/A/R)	L	55/55/56; 42/55/59; 45/50/54; 69/65/65; 40/45/41; 48/52/56; 69/69/73; 59/63/72; 57/53/64	29/29/40; 52/34/49; 50/37/43; 52/31/40; 49/35/41; 39/36/45; 50/37/42; 45/30/34; 48/31/44
	R	45/45/44; 58/45/41; 55/50/46; 31/35/35; 60/55/59; 52/48/44; 31/31/27; 41/37/28; 43/47/36	71/71/60; 48/66/51; 50/63/57; 48/69/60; 51/65/59; 61/64/55; 50/63/58; 55/70/66; 52/69/56

L left, R right, F fixed-feature, A adjusted-feature, R recommended-feature

Table 5 demonstrates the results of the confusion matrix analysis in the aspect of positive/negative predictive value, sensitivity, specificity, and accuracy (in percentage). As in the cross validation, the selection accuracies in the adjusted-feature cases were higher than those in the fixed-feature cases for all smearing conditions on average, and the accuracies of the recommended-feature cases fell between those of the adjusted-feature and those of the fixed-feature cases for most smearing conditions.

Table 5.

Values of positive predictive value (PPV), negative predictive value (NPV), sensitivity (SEN), specificity (SPE), and accuracy (ACC) for 3 feature settings (F/A/R)

	SYM test (F/A/R)			ASYM test (F/A/R)
	NH/NH	ML/ML	HL/HL	NL/ML	NL/HL	ML/NL	ML/HL	HL/NL	HL/ML
PPV	65.5/65.5/58.3	44.7/61.8/54.6	47.4/57.5/55.7	57.0/67.7/61.9	44.9/56.3/50.0	55.2/59.1/55.4	58.0/65.1/63.5	56.7/67.7/67.9	54.3/63.1/59.3
NPV	61.2/61.2/57.7	45.3/59.5/55.4	47.6/55.8/55.3	60.8/66.3/63.2	45.9/54.2/50.0	54.0/57.1/55.6	61.7/67.0/68.2	57.3/65.4/70.2	54.7/59.5/60.9
SEN	55.0/55.0/56.0	42.0/55.0/59.0	45.0/50.0/54.0	69.0/65.0/65.0	40.0/45.0/41.0	48.0/52.0/56.0	69.0/69.0/73.0	59.0/63.0/72.0	57.0/53.0/64.0
SPE	71.0/71.0/60.0	48.0/66.0/51.0	50.0/63.0/57.0	48.0/69.0/60.0	51.0/65.0/59.0	61.0/64.0/55.0	50.0/63.0/58.0	55.0/70.0/66.0	52.0/69.0/56.0
ACC	63.0/63.0/58.0	45.0/60.5/55.0	47.5/56.5/55.5	58.5/67.0/62.5	45.5/55.0/50.0	54.5/58.0/55.5	59.5/66.0/65.5	57.0/66.5/69.0	54.5/61.0/60.0

Unit = %

F fixed-feature, A adjusted-feature, R recommended-feature

Figure 3 demonstrates the results of a non-parametric Mann–Whitney test (two-sided test) for the BCI accuracies of the cross validation in fixed-feature cases. There were significant differences between the no-smearing condition and both of the smearing conditions (p < 0.05) in the SYM test; in contrast, there were no significant differences between smearing conditions in the ASYM test.

Fig. 3.

Results of non-parametric Mann–Whitney test (two-sided test) for the BCI accuracies of the cross validation in fixed-feature cases. a SYM test; b ASYM test

Figure 4 demonstrates the results of a non-parametric Mann–Whitney test in the adjusted-feature cases. There were no significant differences in the results between the test conditions in the SYM test; in contrast, there were significant differences in several comparisons in the ASYM test: [NL/ML vs. NL/HL], [NL/ML vs. ML/NL], [NL/HL vs. ML/HL], [NL/HL vs. HL/NL], [ML/NL vs. ML/HL], and [ML/NL vs. HL/NL] (p < 0.05).

Fig. 4.

Results of non-parametric Mann–Whitney test (two-sided test) for the BCI accuracies of the cross validation in adjusted-feature cases. a SYM test; b ASYM test; *indicates a significant difference (p < 0.05)

Figure 5 demonstrates the results of a non-parametric Mann–Whitney test in the recommended-feature cases. There were no significant differences in the results between the test conditions in the SYM test; in contrast, there were significant differences in several comparisons in the ASYM test: [NL/HL vs. ML/HL], [NL/HL vs. HL/NL], [ML/NL vs. ML/HL], and [ML/NL vs. HL/NL] (p < 0.05).

Fig. 5.

Results of non-parametric Mann–Whitney test (two-sided test) for the BCI accuracies of the cross validation in recommended-feature cases. a SYM test; b ASYM test; *indicates a significant difference (p < 0.05)

Figure 6 demonstrates the results of comparisons between fixed-feature, adjusted-feature, and recommended-feature cases. In the SYM test, there were no significant differences between different feature settings for the no-smearing and high-level smearing conditions; in contrast, there were significant differences between the adjusted-feature and recommended-feature cases for the mid-level smearing condition. In the ASYM test, there were no significant differences between different feature settings for the NL/ML, NL/HL, ML/NL, and ML/HL conditions; in contrast, there were significant differences between the fixed-feature and recommended-feature cases for the HL/NL condition, and there were also significant differences between the fixed-feature and adjusted-feature cases and between the fixed-feature and recommended-feature cases for the HL/ML condition.

Fig. 6.

Results of comparisons between fixed-feature, adjusted-feature, and recommended-feature cases. a SYM test; b ASYM test

Discussion

Several previous reports focused on the effect of psychological statuses on the auditory-evoked potential (AEP) characteristics (Voicikas et al. 2016), but most of them did not further investigated whether such AEP variations significantly affect the accuracy of a BCI system, which we did in this study. In addition, in those previous studies, the authors only observed the AEP variations due to the psychological factors; however, in this study, we also verified that the negative effect of such AEP variations on the accuracy of BCI system can be reduced by carefully-adjusting the feature settings of the BCI algorithm, which is the key significance of our study.

In the SYM test, the average BCI accuracies (when considering both left- and right-concentration cases) were 63.0, 60.5, and 56.5% for the no-smearing, mid-level smearing, and high-level smearing conditions, respectively, in the adjusted-feature cases, but were 63.0, 45.0, and 47.5% for the same conditions in the fixed-feature cases. This result implies that, when the smearing levels of the left/right ears are similar to each other, selection accuracy decreases significantly as the level of smearing becomes higher (that is, as the bandwidths of the ROEX filter in the vocoder become wider) if the feature setting is fixed regardless of the smearing level variations. However, if the feature setting is fined-tuned according to the smearing level variations (as in the adjusted-feature and recommended-feature cases), the decrease in selection accuracy is significantly reduced (Figs. 3a, 4a, 5a). In addition, in the ASYM test for the adjusted-feature cases, the average selection accuracies were relatively higher when subjects were concentrating on the direction of the lower smearing level (that is, the bandwidths of ROEX filter in the vocoder were narrower) in four of six conditions (ML/NL, ML/HL, HL/NL, and HL/ML) (Table 3); more specifically, the accuracies were 12% higher in the ML/NL setup, 6% higher in the ML/HL setup, 7% higher in the HL/NL setup, and 16% higher in the HL/ML setup. This result implies a tendency in which the selection accuracy becomes higher when the auditory stimulus is more easily recognized. These experimental results suggest that (1) the spectral smearing of the stimuli due to the widened auditory filter bandwidth affect the performance of the auditory BCI system, and (2) such a decrease in system performance can be seriously reduced by fine-tuning the settings of internal features according to the degree and pattern of left/right smearing of the individual. However, this tendency was not consistent in the case of recommended-feature; that is, the accuracies were relatively higher when subjects were concentrating on the direction of the lower smearing level in two of six conditions (NL/ML and ML/HL). We cannot explain the reason for this experimental result at this point, but the important thing is that the average BCI accuracies in recommended-feature cases were greater than those in fixed-feature cases in most conditions.

Generally, the selection of a proper feature set critically affects the performance of a classification algorithm. When prior knowledge about an optimal feature set is already given, it is just necessary to extract a decision boundary that minimizes training error by applying several classification tools. However, in reality, information about the optimal feature set among numerous possible candidates is not given in most situations. In several studies, tens of features were naively applied to a classification algorithm (Lee et al. 2016); however, an increase in feature dimensions is generally accompanied by algorithm complexification, increased computational load, and cost increase. For this reason, in many studies, measurements of ASSR amplitudes at MFs were directly compared to each other for selection, or feature optimization techniques, such as principal component analysis and linear discriminant analysis, were utilized to reduce the required features (Nakamura et al. 2013; Kim et al. 2011). In this study, we limited the number of features to three, considering those requirements and the report of Kim et al. (2011), and used the three features that showed maximal left/right differences between left-concentration and right-concentration cases as inputs for classification. The results of the cross validation and confusion matrix analyses demonstrated that (1) the optimal composition of features is affected by the auditory filter characteristics of an individual, and (2) a reduction in BCI performance due to the deteriorated filter characteristics can be counteracted by the fine-tuning of features, which implies that the clinical benefits of a BCI system can be improved by considering the pathophysiological status of the sensory organs of the recipient. However, additional investigations should be required to verify whether the same trend is observed in other types of BCI algorithms.

For the current results to be applied to a real, severely motor-disabled patient who needs auditory BCI, additional information about the degree and the pattern of hearing impairment is necessary. If such data are already provided, feature fine-tuning can be performed using that data; if not, additional audiometric assessment is essential to acquire the required data. However, conventional test protocols, such as pure-tone audiometry, request the voluntary co-operation of the subject (i.e., raising a hand if a signal is audible); thus, it is not applicable to severely motor-disabled patients. In accordance with the previous reports, the ASSR threshold is highly correlated with the audiogram pattern of an individual with sensorineural hearing impairment (Lin et al. 2009), which implies that the degree and pattern of hearing impairment can be assessed by analyzing the characteristics of an ASSR signal. Therefore, the following steps may be necessary for actual application: (1) extract information about the patient’s auditory filter characteristics by ASSR threshold measurement; (2) determine the proper feature set according to the results of audiometric assessment (e.g., utilizing Table 2); and (3) extract optimal decision boundaries for the selected feature set by initial training. However, as shown in Table 2, the primary features are different with each subject, even when the degree and the pattern of smearing are the same, which implies that information about the degree and the pattern of hearing impairment are not sufficient for specifying an optimal feature set for a specific subject. To ameliorate this problem, the concept of a recommended feature set was introduced in this study, and our calculation results demonstrated that applying such a recommended feature set to all subjects could result in improved BCI accuracies for smearing conditions compared to those from fixed-feature cases. However, it is also necessary to perform more clinical tests using extensive numbers of hearing-impaired subjects with various symptoms and of different ages in future studies.

There are several clinical symptoms of sensorineural hearing impairment, such as a narrowed dynamic range, diminished temporal/frequency resolution, dead zone, loudness recruitment, and tinnitus (Moore 2003). Among these, the effects of a narrowed dynamic range are not so serious in BCI applications, because the sound level of a stimulus can be adjusted to a comfortable level for a subject if the recognized sound level is weak. In addition, the effect of a dead zone or extremely high hearing threshold at a specific frequency area can also be evaded by adjusting the carrier frequencies of stimuli. Therefore, in this study, we mainly focused on the symptom of spectral smearing due to the deteriorated temporal/frequency resolution.

Pathophysiological statuses of the ear and the intracranial audio-processing areas, such as the brainstem, primary auditory cortex (Tanaka et al. 2013; Galambos et al. 1981), and gyrus (Saupe et al. 2009), may affect the brain response (e.g., evoked potential) to a specific auditory stimulus; however, several reports have revealed that long-term sensorineural hearing impairment can affect the morphology, structure, and connectivity of audio-sensing intracranial structures (Zhang et al. 2015), e.g., a decrease in white matter volume and altered white matter microstructure in Heschl’s gyrus (Emmorey et al. 2003; Smith et al. 2011); increased functional connectivity in the right posterior frontal lobe, right precentral gyrus, right supramarginal gyrus, right cuneus lobe, and right superior frontal gyrus (Li et al. 2015); abnormal functional connectivity of the default mode network and cortical reorganization (Li et al. 2015); and increased amplitude of low-frequency fluctuation in the right inferior and middle temporal gyrus (Yang et al. 2014). In addition, there was a report that demonstrated the effect of an injured site in the brain due to hearing impairment on the characteristics of an ASSR signal (Pauli-Magnus et al. 2007). Considering those reports, it is necessary to perform additional simulations and clinical tests using extensive hearing-impaired subjects with various impairment characteristics and injury sites to more accurately investigate the effect of sensorineural hearing impairment on the performance of an auditory BCI system (Strauss et al. 2010). Furthermore, modification in the intracranial structure is a commonly observed symptom in patients with Parkinson’s disease (Yi et al. 2017; Yuvaraj and Murugappan 2016; Han et al. 2013); thus, the relationship between the Parkinson’s disease and performance of BCI system would also be an another topic to be investigated.

In this study, we synthesized various smearing-reflected stimuli using an 8-channel vocoder. During implementation of the vocoder, the levels of mid- and high-smearing were determined by considering the moderate and severe hearing loss conditions suggested in previous reports (Glasberg and Moore 1986; Baer and Moore 1994). However, waveform differences between the mid- and high-smearing setups were not prominent, as shown in Figs. 1 and 2, which might result in a less marked difference between the two smearing conditions in the SYM test (Fig. 3a).

The limitations of the current study were that (1) the effect of intracranial modification due to long-term sensorineural hearing impairment was not evaluated because only normal-hearing subjects participated in the clinical tests, and (2) the effect of aging on the brain response to acoustic stimulus was not evaluated because only young subjects participated in the test (Profant et al. 2015; Tomoda et al. 2012).

Conclusion

In this study, we investigated the effects of spectral smearing on the characteristics of evoked ASSR signal and on the performance of the ASSR-based BCI algorithm. Results of the study demonstrated that spectral smearing of auditory stimuli can affect the amplitude of the evoked ASSR signal and also decrease the performance of the BCI algorithm, and that such negative effects of spectral smearing on the BCI accuracy can be reduced by adjusting the feature settings of the BCI algorithm on the basis of the results acquired a posteriori. We think that these results have a potential to improve clinical benefits of a BCI recipient with sensorineural hearing impairment.

Electronic supplementary material

Below is the link to the electronic supplementary material.