Improving N1 classification by grouping EEG trials with phases of pre-stimulus EEG oscillations

Abstract

A reactive brain-computer interface using electroencephalography (EEG) relies on the classification of evoked ERP responses. As the trial-to-trial variation is evitable in EEG signals, it is a challenge to capture the consistent classification features distribution. Clustering EEG trials with similar features and utilizing a specific classifier adjusted to each cluster can improve EEG classification. In this paper, instead of measuring the similarity of ERP features, the brain states during image stimuli presentation that evoked N1 responses were used to group EEG trials. The correlation between momentary phases of pre-stimulus EEG oscillations and N1 amplitudes was analyzed. The results demonstrated that the phases of time–frequency points about 5.3 Hz and 0.3 s before the stimulus onset have significant effect on the ERP classification accuracy. Our findings revealed that N1 components in ERP fluctuated with momentary phases of EEG. We also further studied the influence of pre-stimulus momentary phases on classification of N1 features. Results showed that linear classifiers demonstrated outstanding classification performance when training and testing trials have close momentary phases. Therefore, this gave us a new direction to improve EEG classification by grouping EEG trials with similar pre-stimulus phases and using each to train unit classifiers respectively.

Introduction

A brain-computer interface (BCI) is a communication or control system that provides the brain with a new, non-muscular channel to convey messages and commands to computers (Wolpaw 2010). To establish this channel, electroencephalography (EEG) has been widely used to convey users’ intent. EEG activity recorded from the scalp appears to be a promising signal for brain-computer communication. Those EEG based BCI recognizes specific features in EEG signals, including visually evoked potential (VEP), slow cortical potential (SCP), dynamics of brain oscillations event related synchronization/desynchronization (ERS/ERD), and some Event Related Potential (ERP) components (Bin et al. 2009; Guido et al. 2007; Sannelli et al. 2010; Hoffmann et al. 2008).

EEG based BCI, such as the P300-based virtual keyboard and the SSVEP (Steady state VEP)-based phone-dialing system, are well-known reactive BCIs, which derive their output commands from brain activity in response to external stimulation (Zander and Kothe 2011). In these BCIs, users code their intent into the well-designed stimulus that elicits a particular ERP response and the computer decodes the user’s intent from the measured brain response. In addition to the P300 and SSVEP, another ERP component named N1 was also studied in regard to the classification of visual stimulus-evoked ERP (Wang et al. 2012). N1 is a large negative component appearing approximately 150–210 ms after a visual stimulus. Specifically, a N1 evoked by a face stimulus is termed the N170 component. Human faces can evoke larger N170 ERP components than non-face objects (Itti 2002). Neuroscience researchers revealed that N1 reflects the perception and processing of low-level visual features inside our brain (Antal et al. 2000). In the domain of EEG-based object discrimination, the spatio-temporal activation patterns in N1 components when participants perceived four different categories of visual objects (faces, buildings, cats and cars) was intensively studied (Wang et al. 2012). Some recent studies further showed that it is possible to discriminate visual objects using N1 components in a single-trial EEG (Xu et al. 2008; Shenoy and Tan 2008).

A typical application of N1-based BCI is image categorization, where EEG signals are acquired as users viewing faces or non-face images, and then N1 features are extracted and fed into the classifiers to predict the category of the image. An N1-based image categorization system, as a reactive BCI, suffered from the trial-to-trial variation in the ERP features. In fact, the amplitudes and latencies of the ERP components in a single-trial EEG are quite variable over a long-term time scale and this inconsistency degenerates the generalizability of EEG classifiers in reactive BCIs. To improve the generalizability of the classifiers for a single-trial EEG, it is reasonable to encourage users to control their brain signals to produce stable ERP and thus increase the correlation between EEG features and user intent (Wolpaw et al. 2002). However, it was difficult to train an excellent subject who could proficiently control his/her brain signals. To relieve the inevitable trial-to-trial variation in ERP components due to fatigue and changes in task involvement during the experiments, adaptive algorithms, statistical signal processing and machine learning methods help to improve the classification accuracy of a single-trial EEG (Blankertz et al. 2011). The adaptive classifier proposed by Shenoy corrects the bias between calibration and feedback sessions in combination with an offline feature selection (Shenoy et al. 2006).

The ensemble of classifiers approach was another effective method to cope with variability. In Rakotomamonjy’s method, each classifier is composed of a linear Support Vector Machine (SVM) trained on a small part of the available data (Rakotomamonjy and Guigue 2008). They grouped consecutive EEG trials in a short term and used them to train unit classifiers. In our N1 classification study, we grouped EEG trials in training dataset according to their pre-stimulus EEG oscillations, and then trained unit classifiers from each group. In other words, instead of utilizing the class discriminatory information of evoked N1 responses to group the EEG trials directly, we group EEG trials by measuring the diversity of the category-related N1 features from the EEG oscillations.

The relation between pre-stimulus EEG and post-stimulus EEG is widely studied recently. Maeder has assessed the influence of power (amplitudes) feature of pre-stimulus SMR on classification of post-stimulus EEG under left/right hand imagery (Maeder et al. 2012). Aside for power features, time–frequency (oscillation phase) features of pre-stimulus EEG was also reported related to sensory-cognitive variables, and recent work reveals that the dynamic oscillation phase can carry information about cognitive processes and sensory representations to a greater degree (Ng et al. 2012). Pre-stimulus EEG oscillations predicted, at least in part, the users’ visual perception ability (Lange et al. 2012; Dugue et al. 2011). Such as that the gamma band activity preceding an incoming stimulus is assumed to modulate the detection of visual objects embedded in noise (Salari et al. 2012). In this sense, the processing of visual stimulus within brain is related to the oscillation of ongoing EEG, and thus the amplitudes and latency of N1 could also be predicted, to some extent, by EEG oscillation. A series of recent literatures even revealed that ERP should not be looked as a deterministic evoked activity embedded in background EEG, but considered as a partial output of the ongoing oscillations of EEG (Qia and Di 2011; Fellinger et al. 2011; Gruber et al. 2004).

Latest studies showed that large-scale neural activation dynamics, observable as EEG rhythms and coupling between these rhythms, is predictive of subsequent behavior. Analysis of coherences for delta, theta and alpha frequency ranges showed the coherence to target responses were higher than other targets, supporting that oscillatory responses of P300 are composed of mainly delta and theta responses (Gu¨ntekin and Basar 2010). Some study showed that low-frequency oscillations(3–12 Hz) played multifaceted roles in movement and memory related tasks and the low-frequency oscillations showed significant coherence with oscillations in other brain regions underlying different cognitive processes (Ekstrom and Watrous 2014).

These studies proved that not only ERP components are correlated with EEG oscillations, but also there is a significant diversity in N1 features for EEG trials with different EEG oscillations. Therefore, considering the variability of N1 features and group EEG trials by measuring their pre-stimulus oscillation, provides a new way toward improving the single-trial EEG classification accuracy (Kuncheva and Whitaker 2003; Sollich and Krogh 1996).

The novel grouping approach was also enlightened by the fact that electrical neural oscillations due to periodic fluctuations of the local electrical field and the intrinsic excitability of neuronal populations, brought out periodic perceptual and attention sampling phenomena (Busch et al. 2009). Busch’s recent work revealed that the perception to visual stimulus was periodically modulated along with ongoing EEG oscillations at approximately 7 Hz and 224 ms before stimulus onset. Thus stimulus presented under different ongoing EEG oscillations, may be processed in a different way in brain perception system and evoked different N1 response (Busch and VanRullen 2010). Busch’s work provided the foundation of grouping EEG trials.

In the present study, we first investigated the correlation between N1 amplitudes and momentary phases of ongoing EEG oscillations before stimulus onset at a single-trial level. As a typical early ERP component, N1 reflects the image discrimination process. The correlation between N1 amplitudes and momentary phases revealed the time–frequency point whose oscillation phase is conducive for image perception and discrimination. Secondly, after we defined the time–frequency point where EEG oscillations are significantly correlated with N1 amplitudes, all of the EEG trials were divided into 10 bins according to their momentary phase. We delineated the performance of linear classifiers trained by the EEG trials in different bins. Our finding provided a novel method to group EEG trials with similar classification features and improve single-trial EEG classification performance in reactive BCI systems.

Materials and methods

Ethics statement

The present study was approved by and conducted in accordance with the Emory University Medical School’s Institutional Review Board on experimental ethics. Prior to the experiment, all of the subjects provided written informed consent.

Subjects and tasks

Eight healthy right-handed subjects with normal or corrected-to-normal vision (five females and three male, 22–28 years old) participated in this study. Data of two subjects were not reported here after re-check their EOG, because we found the EEG of these two subjects were contaminated with too many EOG artifacts by visual inspection.

During the experiment, the participants sat in a quiet dark room with their arms relaxed. The experiment consisted of 10 sessions. During each session, 80 visual stimulus images consisting of 40 faces and 40 buildings were presented to the subjects in a random order in the center of screen. Some of the face images were collected from a face image database (http://pics.psych.stir.ac.uk/) and the other facial pictures were obtained from our personal collection. All 40 faces (20 male, 20 female) were in a frontal pose with a neutral emotional expression and the hair and ears were masked out. The images of buildings were collected from the Internet. Images from both categories were converted to gray-scale, cropped to 300 × 300 pixels, and were placed in the center of gray backgrounds with different orientations. The luminance was manually adjusted to avoid extremely dark or bright images. The average values of global luminance of faces and buildings are 137.38, 138.32 respectively.

Each image presentation lasted 500 ms and was followed by a blank screen and an inter-stimulus interval (ISI) ranging randomly from 850 ms to 1,450 ms. Subjects were asked to concentrate on the fixation point (a red cross) in the center of the stimulus images and the blank screens. Subjects were instructed to avoid making explicit decisions about the category information of the stimulus. The red cross turned blue 10 % of the time throughout the experiment and the subjects were asked to count the number of blue crosses. At the end of each session, the subjects pressed a button on the response box to select a range option containing the counted number of blue crosses. Offline analysis results showed that all of the subjects were able to count the blue crosses correctly.

Recording and preprocessing procedure

The ERP data for the six participants was collected from a 64-channel Brain Products system (including 63 EEG channels and 1 ECG channel; FCz was used as the reference channel) with a sampling rate of 500 Hz. The electrodes were distributed in accordance with the international 10-20 system of electrode placement. We selected 38 of the 63 EEG channels for further processing. The ECG channel and 26 EEG channels (FP1, FPz, FP2, AF7, AF3, AFz, AF4, AF8, F7, F5, F3, F1, F_z, F2, F4, F6, F8, FT7, FC5, FC3, FC1, FCz, FC2, FC4, FC6 and FT8) located in the frontal area that were susceptible to contamination by eye blinks were removed. The EEG data were continuously recorded and was down sampled to 250 Hz. The EEG data were then baseline-corrected and epoched by stimulus conditions. The baseline-correction and epoch processing were performed using EEGLAB, a Matlab-based toolbox.

A total of 400 trials for each category per subject were selected. And the post-stimulus part (from 0 to 700 ms) was band-pass filtered using a causal finite impulse response (FIR) filter (45 orders, 0.15–40 Hz, linear-phase shift). We employed wavelet Analysis to extract pre-stimulus EEG (−800 to 0 ms) time–frequency features, and we mirrored the pre-stimulus part of the EEG at the stimulus onset before wavelet analysis, to avoid the affect from post-stimulus information (Salari et al. 2012). Here, wavelet analysis was employed to estimate the momentary phase for each point in the time–frequency space from −800 to 0 ms and from 4–35 Hz, which means that it used for extracting phase of pre-stimulus signal. The length of the wavelets increases linearly from 1.5 cycles at 4 Hz to 5 cycles at 35 Hz. This means that the length of Morlet wavelet is 3/8 s for 4 Hz, and linearly decreases to 1/7 s for 35 Hz. This modified wavelet transform was selected to optimize the trade-off between temporal resolution at lower frequencies and stability at higher frequencies. At each time and frequency point, the result of the wavelet transformation for the k^th trial is a complex number. The parameter A represents the amplitude of the signal, and φ denotes its phase in Eq. (1).

$$x(t)=A(t,f)e^{i\mathit{\phi }(t,f)}$$ 1

The Morlet wavelet was used as in Eq. (3), where f represents the frequency related to this wavelet and c is the cycle number for frequency f, and fs is the sample frequency of the EEG signals (250 Hz in our EEG data).

$$\begin{array}{cc}& W(f,c,t)=\sqrt{2\mathit{\pi }}{e}^{i2\mathit{\pi }x}\hfill \\ \hfill & \ast \left(e^{(-x^2/2s^2)}-\sqrt{2}{e}^{\left(\left(-x^2/s^2\right)-{(2\mathit{\pi })}^2\ast s^2/4\right)}\right)\hfill \\ \hfill & x=t\ast f/f_s,s=c/5\hfill \\ \hfill & t\in \left[-c\ast f_s/(2f),c\ast f_s/(2f)\right]\hfill \\ \hfill \end{array}$$ 2

Phase bifurcation index

We hypothesized that the EEG trials that were easily classified (the correctly recognized categories by classifiers) or hard to classify (incorrectly classified) were each associated with a particular phase of the momentary EEG oscillations before the stimulus onset. The phase concentration value (or the phase-locking factor) is measured by the inter-trial coherence (ITC), defined as (3) in Busch’s study (Gu¨ntekin and Basar 2010). The value ITC is between 0 (absence of synchronization across trials) and 1 (perfect synchronization). ITC considers only the phase information of each trial in the specific time and frequency point.

$$IT{C}_{(t,f)}=\frac{1}{k}\left|\sum _{n=1}^k{e}^{i({\mathit{\phi }}_{k(t,f)})}\right|$$ 3

To quantify the differences in the phase distributions for the correctly and incorrectly classified EEG trials, a phase bifurcation index (ϕ) was employed to compare the ITC of the correctly and incorrectly classified EEG trials against the ITC of all of the trials:

$${\mathit{\Phi }}_{t,f}=(IT{C}_{correct(t,f)}-IT{C}_{all(t,f)})\times (IT{C}_{incorrect(t,f)}-IT{C}_{all(t,f)})$$ 4

The range of the phase bifurcation index is (−1, 1). A positive phase bifurcation index value means that phases are locked to different phase angles for both of the conditions, while a negative value indicates that only one condition exhibits phase locking.

Linear classifier

The waveforms within the time interval (150–210 ms) of N1 in single-trial ERP responses were averaged along time and were treated as features for classification. N1 features represented by $\mathbf{x}\in {\text{R}}^{\text{N}\times 1}$ (N = 50, the number of selected channels). N1 features here was also called spatial feature as it contained only one sample in a time domain. In this paper, N1 was evoked by two categories of objects (faces and buildings). Therefore, we employed a binary classifier to investigate how well the visual objects could be discriminated.

We employed Fisher Linear Discriminate Analysis (Fisher-LDA) algorithm to classify the features of the ERP components in individual subject level. The LDA for binary problems follows the optimal projection vector w, corresponding to the largest J values (Blankertz et al. 2011; Duda et al. 2001).

$$J=\frac{w^T{S}_{b}w}{w^T{S}_{w}w}$$ 5

The within- and between-class covariance matrix in (5) can thus be defined as.

$$S_w={\sum }_{}^{+}+{\sum }_{}^{-},S_b=({\mathit{\mu }}^{+}-{\mathit{\mu }}^{-}){({\mathit{\mu }}^{+}-{\mathit{\mu }}^{-})}^T\underset{}{}$$ 6

Here, Si is the covariance matrix for the ith class (i = 1, 2), ${\mathit{\mu }}_i\in {\mathbb{R}}^{N\times 1}$ is the mean of the temporally averaged ERP features from the ith class, and u is the global mean of the ERP features from both classes. It is worth noting that both the covariance matrixes and the mean values were computed using samples from the training datasets of each subject.

The projector w was estimated by solving a generalized eigenvalue problem. The first w is an eigenvector satisfying S_bw = λS_ww with the largest eigenvalue λ. Subsequently, the ERP features x from the testing datasets were projected into the feature space defined by w, where the resulting scalar y was used to predict to which category the EEG features belonged. And EEG trials was classified according to the sign of y value.

$$y=w^{t}x+b$$ 7

Results

Results of the phase bifurcation index

The analysis focused on the correlation between EEG phases and the discrimination contained in the EEG trials that can be used by classification. Here, we compared the EEG oscillation phase between the correctly and incorrectly classified EEG trials. We investigated whether the ongoing EEG phase before the stimulus presentation influences the classification by computing the phase bifurcation index.

Each EEG trial was classified with a classifier trained with independent samples. We conducted cross-validation analysis using the EEG samples in the 10 sessions. In the cross-validation analysis, the 10 sessions were divided into five groups (two sessions per group), and the classifier model was trained with four groups and tested with the remaining group. This procedure was repeated until all groups were classified (5-fold cross-validation). According to the classification results of the cross-validation analysis, the EEG trials were separated into correctly and incorrectly classified EEG datasets. The average classification accuracy across subjects was 0.7032 with the standard error 0.002.

Phase bifurcations as averages across electrodes and time points in the 800-ms pre-stimulus interval were computed. The time labeled “zero” denotes the presentation of the stimulus onset. The frequency range from 3 to 100 Hz was analyzed using a continuous wavelet transform of a single-trial EEG, but for increased visibility, the results are plotted only in the frequency range between 4 and 35 Hz because no significant effects were found beyond 35 Hz. The very low frequencies below 4 Hz were omitted due to their sensitivity to outside noise (Fig. 1).

Fig. 1.

The varying period of the Morlet wavelet. The length of the wavelets increase linearly from 1.5 cycles at 4 Hz to 10 cycles at 35 Hz. A is the 1.5-cycle wavelet at 4 Hz and B is the 10-cycle wavelet at 35 Hz. These varying cycles were selected to optimize the trade-off between temporal resolution at lower frequencies and stability at higher frequencies

The phase bifurcation spectrum (averaged across channels and time points in the pre-stimulus interval) showed that the N1 classification accuracy was strongly correlated in the blue time–frequency region (Fig. 2a). Here negative bifurcation index means that one condition is phase locked, while other condition is random phase. For example, in the point about 5.3 Hz and -0.3 s preceding stimulus onset, the ITC value for correct/wrong classified trials is 0.076/0.019, and the ITC value for all trials is 0.021.

Fig. 2.

a Phase bifurcation index (Φ), averaged across all channels and subjects; b The sum of leave-one-out FDR test results of phase bifurcation index (Φ) of all subjects across all channels

We calculated the phase bifurcation spectrum in the black dotted box (about from 4 Hz to 10 Hz and from −0.4 ms to −0.1 ms) in Fig. 2a for other six times, and isolated one subject each time. We also analyzed the resulting distributions of the P values with the false discovery rate (FDR) with a 1 % expected rate of falsely rejected null hypotheses to correct for the effect of multiple comparisons(Benjamini and Hochberg 1995) of every phase bifurcation result. We added all of the six FDR results together to locate the time–frequency region, in which all of the six P values are significant (Fig. 2b).

The phase bifurcation effect in this time–frequency range had anterior-parietal topography with the top value at channel POz (Simanova et al. 2010) which was reported connecting with N170. The white marker in Fig. 3a indicates the channel of the region of interest in this study, and the subsequent analysis used EEG signals in the channel. We also calculated the phase bifurcation index value of this channel in the time–frequency region from 4 to 10 Hz and from −0.4 to −0.1 ms. The leave-one-out FDR test result is shown in Fig. 3b. We can see in Fig. 3b that the point about 5.3 Hz and −0.3 s preceding stimulus onset demonstrated maximum significance, this point is always significant every time leaving out a different subject.

Fig. 3.

a The distribution of Φ of preceding stimulus onset each channel averaged across all subjects. b Statistical significance of phase bifurcation index (Φ), averaged across all subjects and the channel POz

We analyze the P value with a randomization test based on surrogate data, as previously described (Gu¨ntekin and Basar 2010). As a first step, we computed a sample of random phase angles with a set size identical to the number of the trials in the real dataset for a given subject. These random phases combined with the actual classification results from the data for this subject were then used to compute the phase bifurcation index. The phase bifurcation index according to this procedure would be due to chance. This procedure was repeated 10,000 times per subject, thus producing for each subject a distribution of phase bifurcation index (Φ) values at each point of the time–frequency space based on random phases under the null hypothesis. In a second step, we drew one pseudo-phi value at random for each subject and averaged across all six subjects. By repeating this second procedure 50,000 times, we obtained a distribution of grand averaged Φ values based on random phases. For each time–frequency point, the statistical significance of the Φ observed in the real data was computed as the proportion of random Φ that exceeded the observed Φ.

To investigate the influence of pre-stimulus phases on the N1 component, we depicted N1 classification accuracy as the function of pre-stimulus phases at 5.3 Hz −0.3 s and 5.3 Hz −0.296 s. We segmented the phase from −π to +π into six bins and assigned each trial to one of the six bins according to its phase. The trial number and the phase range in each bin are shown in Table 1. The number of EEG trials recorded as the subjects viewed images of the faces or the buildings is listed in the format “number of face images/number of building images”. The phase ranges of each of the six bins were equal, and the number of EEG trials from the face or the building stimuli was not significantly different. The results show that although the stimuli were presented at random without consideration of the momentary phases, the momentary phases of the EEG trials were almost uniformly distributed in the area [−π, π] for both categories. Before the trials were classified, we balanced the number of trials per class in some bins which have more trials in one class than the other class for each subject by removing trials in predominant class randomly until the number difference less than 10 % of the number sum of both classes. The numbers for face and building images in each bin was shown in Table 1.

Table 1.

Numbers of trials for the face images and the building images in each bin

Bin	Phase range	Sub 1	Sub 2	Sub 3	Sub 4	Sub 5	Sub 6
1	[−π, −2π/3)	77/73	62/66	60/69	71/71	68/71	70/69
2	[−2π/3, −π/3)	69/65	75/71	64/54	59/66	67/72	65/59
3	[−π/3, 0)	55/48	67/64	58/49	52/59	60/58	56/60
4	[0, π/3)	78/88	75/68	69/77	73/67	74/75	78/69
5	[π/3, 2π/3)	64/59	63/70	70/76	76/77	66/60	68/78
6	[2π/3, π)	57/67	58/61	79/75	69/60	65/64	63/65

Numbers are shown on the left/right side

We are also interested in the fluctuation of the post-stimulus ERP component (N1) amplitudes over pre-stimulus phases. To investigate the influence of the pre-stimulus phases on the N1 amplitudes, we plot N1 amplitudes as the function of pre-stimulus phases at 5.3 Hz and −300 ms. The N1 amplitudes were estimated from the time window 150–210 ms from the stimulus onset at channel POz. POz was selected because N1 has been reported to be very obvious around the occipital lobe (Simanova et al. 2010). The pre-stimulus ongoing EEG phase was computed with wavelet analysis.

Figure 4 shows the averaged N1 amplitudes across subjects in each phase bin. We can see that the N1 amplitudes changed with the momentary phases and that the N1 amplitudes reach their peaks when the phase is approximately the bin with zero phase. The significant phase effect on modulation of the N1 amplitudes was also confirmed by statistical test, F(5,30) = 9.548, p < 0.001.

Fig. 4.

N1 amplitude as a function of the pre-stimulus phase. The amplitudes of N1 fluctuated with the average momentary EEG phases at 5.3 Hz and −300 ms. All of the trials were divided into six bins according to the EEG oscillation phases

Classification of EEG trials with different momentary phases

The discriminative information of the N1 features in each bin was evaluated using the accuracy of a Fisher-LDA. To some degree, the classification accuracies could evaluate the distinctness of the N1 features for the face or the building images. We compared the classification results from the EEG trials in the different momentary phase bins and described the accuracy performance of the LDA classifiers as a function of momentary phases. We also compared the traditional classifiers trained from all EEG trials in all phase bins in training dataset with unit classifiers.

In this study, a 5-fold cross-validation method was used to obtain robust classification results. The EEG trials from the face or building image stimuli in each bin were randomly divided into five equal subsets. Four subsets from each bin were used as training dataset to train an LDA classifier, and the remaining one subset was used as test data. The trained classifier was validated with the one subset in all six phase bins. The averaged classification accuracy of five permutations was used as the final classification accuracy. This process was repeated 50 times, and the mean accuracy of these 50 times is shown in Fig. 4. Accuracy of all unit classifiers and the traditional classifier fluctuated over phases. Statistical test revealed significant phase effect for all classifiers, F(5,294) > 6.322, p < 0.001.

Comparison of the classifiers showed that the classifier trained from bin 4 obtained the best overall classification performance. The classifier trained from bin 3 is most suitable for test trials in phase bins 1 and bin 5. The classifier trained from bin 4 is superior to others for test trials in phase bin 2 and bin 4, and the classifier trained from bin 1 gets the best accuracy for the test trials in bin 3 and bin 6.

Classifier ensemble results

The classification results in Fig. 5 show that there is a significant diversity among the classifiers trained from each phase bin. We also tend to yield a better predictive performance than could be obtained from any of the unit classifiers by the classifier ensemble using the significant diversity among classifiers. Unit classifiers trained from each bin are experts in some local area of the feature space related to the specific momentary phases, and they are combined with the classifier selection. According to the pre-stimulus EEG oscillation phases, we selected the best unit classifier response for the partition of the feature space. The output of the ensemble classifier for the incoming N1 features x is computed as follows.

$$f_{ensemble}(x)=\left\{\begin{array}{c}{f}_{bin3},phase(x)\in bin1,bin5\\ f_{bin4},phase(x)\in bin2,bin4\\ f_{bin1},phase(x)\in bin3,bin6\\ \end{array}\right.$$ 8

Fig. 5.

Classified accuracy in different phase bins

In Eq. (8), x represents the classification features extracted from the N1 components in a single EEG trial, and phase(x) represents the pre-stimulus phase of that trial. The parameter f_{bin i}, where i = 1 to 6, represents the output of the unit classifiers.

To compare the classification accuracy of traditional classifiers and classifiers ensemble based on pre-stimulus EEG phases, we divided all of the EEG trials into a training dataset and a test dataset in ratio of 4:1. And the training dataset was further divided into six phase bins. The EEG trials in each bin are employed to train the unit classifiers. For comparison, we also used all of the EEG trails in the training data to train the classifiers as the traditional methods. The classification accuracy of the test dataset for the ensemble classifier, the best unit classifier from the 3rd bin and the traditional classifier are averaged across all subjects.

We repeated the above process 50 times. Each time we reassigned training and test dataset and computed the averaged accuracy for the ensemble classifier, the best unit classifier and the traditional classifier. Figure 6 showed the classification performance on test datasets. Statistical tests (paired Wilcoxon sign rank test) on classification results show that the difference between the best classifier and the traditional classifier is significant (p < 0.001, 50 pairs) and the difference between the ensemble classifier and the traditional classifier is also significant (p < 0.001, 50 pairs). The difference between ensemble classifier and the best classifier is not significant.

Fig. 6.

The classification accuracy of three different classifiers. **means p < 0.01

Discussion

Pre-stimulus momentary phases and classification features

Busch revealed that detection performance for attended visual stimuli fluctuates over the phase of spontaneous oscillations at 7.1 Hz and −224 ms (Ekstrom and Watrous 2014). In the present study, we also found that the amplitudes of N1 in the single-trial EEG could be predicted to some extent by the pre-stimulus phase at 5.3 Hz and −300 ms. The time point in our study is a little earlier than the time interval of the study performed by Busch, possibly indicating that the visual perception and the brain activity to produce the fluctuation of the N1 components along with the EEG oscillation are at approximately 5.3 Hz.

Considering the fact that N1 is a typical ERP component of visual stimulus processing, the results further validate the hypothesis that pre-stimulus phases partly represent visual perception. An image presented in different moment phase state may activate the visual cortex in different strength or in a different pathway due to the fluctuation in perception and, therefore, the evoked N1 response to the stimulus would contain distinct N1 features. And therefore, the evoked N1 response to the stimulus would also vary with the momentary EEG oscillation phases. The averaged classification results and N1 amplitudes have been demonstrated to share a similar variation trend as the varying EEG phase. Figure 5 shows that the N1 features in those bins close to the zero phases are easy to classify, and almost all of the classifiers trained from the different bins showed a high classification accuracy in those bins with large N1 amplitudes, while the EEG trials in the fifth bins where the N1 amplitudes are small seem difficult to categorize by all classifiers. These findings also validate the above assumption that N1 is the proxy component standing for the image categorization process within our brain. When the EEG oscillated in the right phases, N1 was enhanced, and thus, the classification features in the N1 response to different stimuli (face vs. Building) contained more discrimination information.

Figure 5 also demonstrates that the performance curves for the different classifiers showed different trends, and the classification accuracy for each classifier trained in a bin also changed over the pre-stimulus phases, providing another proof that the classification features in the N1 components fluctuate with the phases of the EEG oscillations, consistent with our assumption that the EEG trials in the different bins may have different classification features, such as N1 amplitudes and activation topography in the scalp. The pre-stimulus phase seems like a promising measurement of the quality of the post-stimulus EEG features. In this sense, the EEG trials have divergent features across the bins. Grouping of the EEG trials with the pre-stimulus phase seems to be reasonable, and employing the EEG groups to train the unit classifiers and then to ensemble their outputs to improve the classification accuracy also seems reasonable.

Session-to-session transfer and classification features

Note that the ERP response to a visual stimulus typically varies strongly across sessions due to different psychological states of the subject. A subject might, for example, show different states of fatigue and attention across sessions. This ERP variation is well known as session-to-session variation in BCI community. To prove that the change of classification accuracy in Fig. 5 is ascribed to the N1 difference derived from brain state fluctuate periodically in very short time (5.3 Hz), not from session-to-session variation in relative long time scale, we examined the influence of non-stationary brain signals on the classification performance of visual image categorization.

Classification accuracy and sessions

Since image stimuli onset in each session is randomized and unpredictable, the phases of EEG oscillations should be randomly distributed across all trials in the sessions. In other words, the trials in each phase bin were not grouped by time, and the change in classification accuracy in Fig. 5 should be ascribed to the N1 difference derived from the brain state fluctuating periodically (represented by EEG phases) in a very short time (approximately 5.3 Hz), not from session-to-session variation on a relatively long time scale.

It’s well known that the ERP response to a visual stimulus, such as N1, typically varies strongly across sessions (session-to-session variation) due to the different brain states of the subjects. A subject might, for example, show different states of fatigue and attention across sessions. To investigate variation in the classification features over different sessions, we further compared the classification accuracy of the EEG trials in the different sessions. All the trials from every two consecutive experimental sessions were pooled together to form a subset. Then we executed a five-fold cross validation for classification (Fig. 7).

Fig. 7.

The cross-validation classified accuracy in different sessions

Figure 7 shows the cross-validation classification results of the different subsets (double sessions). The 5-fold cross-validation method was identical to the method of Fig. 5. The significant session effect on accuracy of classifiers trained from each subset were confirmed by statistical test, F(4,245) > 3.429, p < 0.01.

Figures 5 and 7 further indicated that the N1 featured EEG trials showed a phase-to-phase variation, as well as a session-to-session variation.

The ensemble classifier

One of the most important findings in this study was the phase-to-phase variation of the classification features in the N1 components. This finding provided a novel method to group all of the EEG trials related to a similar phase. In this manner, the EEG trials recorded during a relatively stable visual perception state will be pooled into a group. Ideally, we want to cluster the training signals into several partitions so that each partition has “similar” classification features. At the same time, there is a significant diversity among the partitions. Each group was hypothesized to determine a decision boundary for a different subset of the EEG trials and ensemble the output of the unit classifier trained from each group to help to reduce the classifier variability due to the inherent signal variability among the subsets.

The ensemble classifier based on the phase-to-phase variation is not the same as the ensemble classifier based on the session-to-session variation of the classification features. In the study by Alain Rakotomamonjy, a subset of consecutive EEG trials was grouped to train the unit classifiers. This method grouped the EEG trials in a session time and hypothesized that the brain activities and acquisition run did not change dramatically over time. With regard to the classification of the single-trial EEG signals in test datasets, each unit classifier has an output, and the global output of the ensemble classifiers is defined as the average of all of the single classifiers.

The advantage of grouping EEG trials with the pre-stimulus phases lies in the fact that this scheme is suitable for the on-line application of the EEG classification. Considering the multiple classifiers trained in the different sessions, we cannot ensure that the classification features of a new coming test trial will resemble which training sessions, and thus, we find it difficult to decide which unit classification trained from the sessions is reliable for the coming EEG trial. As for the ensemble classifiers trained with the EEG trials grouped by the pre-stimulus phases, the unit classifier to classify the EEG trials was selected based on the pre-stimulus EEG phase. This scheme virtually divides the feature space into several regions, and each classifier defines the decision boundary of a region. For each test trial, we calculated the pre-stimulus phase to determine in which region the phase was located and determined which classifier was suited for the EEG trial. In this way, for each coming trial, only the optimal classifier will work. In Rakotomamonjy’s ensemble methods, we need the output of all of the classifiers at the same time.