Improved signal processing approaches in an offline simulation of a hybrid brain–computer interface

Abstract

In a conventional brain–computer interface (BCI) system, users perform mental tasks that yield specific patterns of brain activity. A pattern recognition system determines which brain activity pattern a user is producing and thereby infers the user’s mental task, allowing users to send messages or commands through brain activity alone. Unfortunately, despite extensive research to improve classification accuracy, BCIs almost always exhibit errors, which are sometimes so severe that effective communication is impossible.

We recently introduced a new idea to improve accuracy, especially for users with poor performance. In an offline simulation of a “hybrid” BCI, subjects performed two mental tasks independently and then simultaneously. This hybrid BCI could use two different types of brain signals common in BCIs – event-related desynchronization (ERD) and steady-state evoked potentials (SSEPs). This study suggested that such a hybrid BCI is feasible.

Here, we re-analyzed the data from our initial study. We explored eight different signal processing methods that aimed to improve classification and further assess both the causes and the extent of the benefits of the hybrid condition. Most analyses showed that the improved methods described here yielded a statistically significant improvement over our initial study. Some of these improvements could be relevant to conventional BCIs as well. Moreover, the number of illiterates could be reduced with the hybrid condition. Results are also discussed in terms of dual task interference and relevance to protocol design in hybrid BCIs.

1. Introduction

Brain–computer interfaces (BCIs) allow communication without movement. Users can send messages or commands through direct measures of brain activity (Wolpaw et al., 2002). Most noninvasive BCIs measure brain activity through electroencephalographic (EEG) sensors placed on the head (Mason et al., 2007). Four types of noninvasive BCIs have been described in the literature, categorized according to the type of brain activity used for control (Wolpaw et al., 2002; Allison et al., 2007): P300s (Farwell and Donchin, 1988), steady-state evoked potentials (SSEPs) (Middendorf et al., 2000; Müller-Putz et al., 2006), slow cortical potentials (SCPs) (Birbaumer et al., 1999), and event-related desynchronization (ERD) (Kalcher et al., 1996; Pfurtscheller and Lopes da Silva, 1999).

It might also be possible to combine two or more BCIs so that subjects produce two or more of these different brain activities. A BCI could also be combined with a device based on other physiological signals such as breathing or heart rate (Scherer et al., 2007). BCIs might also be integrated with conventional interfaces such as keyboards, mice, or joysticks (Nijholt et al., 2008). More recent work showed that an SSVEP BCI could be combined with a new type of ERD BCI called a “brain switch” (Pfurtscheller and Solis-Escalante, 2009; Solis-Escalante et al., 2010; Pfurtscheller, 2009).

In our recent paper (Allison et al., in press), we introduced a dual task paradigm relevant to a hybrid BCI that combines visual attention and imagined movement. Subjects performed three different tasks: ERD-only (by imagining left or right hand movement to produce ERD); SSVEP-only (by focusing on one of two flickering LEDs to produce SSVEPs); and a hybrid condition with both tasks. This hybrid condition provides a classifier with two different types of brain signals. The additional signal could improve accuracy. This improvement might be especially important for subjects who are not proficient with BCIs based on one of these signals (either ERD or SSVEP), a phenomenon called “BCI illiteracy” by some researchers (Kübler and Müller, 2007).

While the initial study collected data offline and thus did not use a BCI, results seemed promising for future hybrid BCI systems. However, this study left open many questions about how to best analyze and combine ERD and SSVEP signals, and we wanted to explore some of these questions before an online implementation. Indeed, due to the novelty of hybrid BCI systems, many opportunities for improvement have not been investigated. For example, our paper noted that the hybrid condition might have improved accuracy by providing the classifier with more information, and/or by producing EEG activity that was easier to classify.

The principal goal of the present study was to explore eight possible avenues for improving classification accuracy based on the data recorded from our offline simulation of a hybrid ERD/SSVEP BCI (see also Fig. 1).

Fig. 1.

Overview of all eight analyses in this study.

First, we consider improving classification in the SSVEP-only runs by including harmonics of the SSVEP stimulation frequencies. We also assess a harmonic phase coupling (HPC) approach that could also improve SSVEP classification accuracy. Second, we extend the first analysis to the hybrid runs. Third, we simulate an “artificial” hybrid BCI by combining features of the ERD-only and SSVEP-only runs. By comparing this to the original hybrid data, we can determine whether the hybrid condition yielded stronger ERD activity than the ERD-only runs, and stronger SSVEP activity than the SSVEP-only runs. The fourth and fifth analyses assess whether performing two tasks simultaneously affects task performance.

In the sixth analysis, we compare the three conditions (ERD-only, SSVEP-only, and hybrid) using some of the new methods introduced here. The seventh and eighth analyses both assessed the relevance of the feature pool size across the different run types. We wanted to explore the hypothesis that the improvement in the hybrid condition occurred because more features, and/or different feature types, were available to the classifier.

While the main goal was to improve accuracy, we also interpret these results in terms of relevance to dual task interference and simultaneous task performance. Did the addition of a second task (such as a motor task) impair performance in the primary task (such as a visual attention task)? If so, how much? How could hybrid BCIs best combine two or more tasks?

2. Methods

The materials and methods used to collect and analyze the data are briefly reviewed here. We then describe the new signal processing techniques introduced in this study (Fig. 1 presents an overview of all analyses).

2.1. Subjects

Fourteen healthy subjects (six women and eight men, age range 17–31 years) participated in this study. No subjects previously used any BCI. Subjects sat in front of a 17 inch flat screen. Red LEDs were attached below the screen on the left and right side and flickered at 8Hz and 13 Hz, respectively. Fig. 2(a) shows the apparatus for stimulus presentation used in this study.

Fig. 2.

(a) Computer screen and flickering LEDs (below screen) used in this study. (b) Timing of each trial.

2.2. EEG recording

Three bipolar electrodes over the motor cortex (C3, Cz, C4) and two bipolar electrodes over the primary visual cortex (O1, O2) were recorded. A ground electrode was placed on Fpz. All impedances were kept below 5 kΩ. The EEG signals were bandpass-filtered between 0.5–100 Hz, amplified with a g.BSamp amplifier (g.tec OEG, Graz, Austria), and sampled at 250 Hz (16 bit DAQ card National Instruments NI6031E). Sensitivity was set to 50 µV.

2.3. Experimental paradigm

Subjects performed three different tasks. During the first two runs, called ERD-only, subjects imagined moving the left or right hand to produce ERD. Subjects were instructed to imagine moving the hand (first-person movement imagery) instead of imagining observation of hand movement (third-person movement imagery), since the former yields stronger ERD activity (Neuper et al., 2005). During the next two runs, called SSVEP-only, subjects focused on one of two flickering LEDs to produce SSVEPs. During the last four runs, called hybrid, subjects were instructed to perform both tasks simultaneously. During each run, 20 left and 20 right cues in the form of an arrow were presented in random order. Each of these 40 trials proceeded as follows: first, a fixation cross appeared on the screen. After two seconds, the subjects viewed an arrow that pointed either to the left or right. They had to perform the corresponding task until the arrow disappeared after another four seconds. Then, the screen was blank for two seconds, followed by a break of 0.5–1.5 s before the next trial. Subjects did not receive feedback. Fig. 2(b) illustrates the timing scheme of the experimental paradigm.

2.4. Analysis 1: improved SSVEP feature extraction in the SSVEP-only runs using harmonics and harmonic phase coupling

In our first analysis, we addressed whether SSVEP classification could be improved by using more sophisticated feature extraction methods than in our original paper. In a preliminary step, all data sets were visually inspected, and the trials containing muscle artifacts were discarded. Most conditions in this analysis relied on logarithmic band power features with 1Hz bands centered around the stimulation frequencies 8Hz and 13 Hz. Signals from both O1 and O2 were used. The signals were bandpass-filtered, squared, and smoothed with a moving average filter. After that, the logarithms of these signals were used as band power features. Earlier work showed that some subjects produce distinct SSVEP activity at the second or third harmonic in addition to the fundamental frequency (Müller-Putz et al., 2005, 2008; Allison et al., 2008). Our approach in this paper is equivalent to the well-established harmonic sum decision (HSD) approach using three harmonics (Müller-Putz et al., 2005). Accordingly, we extracted features using the fundamental frequencies (first harmonics), second harmonics, and third harmonics. We calculated these band power features with three different smoothing windows (0.5 s, 1 s, and 1.5 s), hereafter called HSD1–HSD3. The size of the feature vector did not depend on the length of the smoothing window; each feature vector comprised 12 features (two electrodes with three bands each).

The fourth condition used a technique called harmonic phase coupling (HPC). Our earlier work showed that this approach improved accuracy in about half of the subjects tested (Krusienski and Allison, 2008). HPC exploits amplitude and phase coupling of the harmonically related sinusoidal components by constructing a matched filter template MF(n) according to the following equation (Krusienski et al., 2007; Krusienski and Allison, 2008):

$$\text{MF}(n)={\displaystyle \sum _{k=1}^N{a}_k\text{cos}\left(\frac{2\pi nk{f}_f}{f_s}+{\mathrm{\Phi }}_k\right)}$$

where n is the template sample number, f_s is the sampling frequency (250 Hz in our case), f_f is the fundamental frequency (or first harmonic) of the template, N is the number of harmonics to be modeled, and a_k and Φ_k are the amplitude and phase of the individual harmonics, respectively. These model parameters can be simply obtained from the FFT spectrum of the user’s normalized characteristic waveform at each frequency as determined by a phase-aligned average.

Each incoming data epoch is circularly convolved for one period of the MF template in order to evaluate the template at discrete phase shifts, essentially determining the optimal phase correlation between the data segment and the template. The square root of the maximum value of the circular convolution, corresponding to the optimal alignment, is the feature for the data segment. The result is a continuous amplitude analysis, similar to that produced by a single frequency bin of a conventional spectral analysis technique.

We determined separate MF templates with N = 3 harmonics for each of the two channels (O1 and O2) at each of the two stimulation frequencies (f_f = 8Hz and 13 Hz) and the corresponding second harmonic of each stimulation frequency (f_f = 16 Hz and 26 Hz), resulting in eight features. We computed the model parameters of each MF template using the trials of the first run with a target stimulation frequency (or second harmonic of this frequency) corresponding to the fundamental frequency of the template.

The fifth condition was taken directly from our first paper for comparison. It uses a feature selection algorithm (sequential floating forward selection, SFFS) to select the optimal feature set consisting of up to five features. The feature pool consisted of eleven non-overlapping logarithmic band power features (width 2 Hz, between 8–30 Hz) for each of the two electrodes O1 and O2.

All five conditions in this analysis are summarized in Table 1. Features extracted from all methods were subjected to a linear classifier (Fisher’s linear discriminant analysis, LDA). In a first step, we determined the optimal 0.5 s time segment to train the classifier with a so-called running classifier (Müller-Gerking et al., 2000). From this optimal time segment, we used five equally-spaced samples every 0.1 s. We employed a 10 × 10 cross-validation procedure to avoid overfitting. The performance measure of the various methods was the maximum of the cross-validated classification accuracy within a trial.

Table 1.

The five conditions in the first analysis. HSD1, HSD2, and HSD3 differ only in the length of the smoothing window. The HPC condition uses harmonic phase coupling, and the SFFS condition is identical to the analysis in our initial paper.

Method	Harmonics	Smoothing window
HSD1	3	0.5 s
HSD2	3	1.0 s
HSD3	3	1.5 s
HPC	3–4	1.0 s
SFFS	Variable	1.0 s

We compared the performance measures of the five different conditions with a one-way repeated measures analysis of variance (ANOVA). We checked the sphericity assumption and performed Greenhouse–Geisser correction when necessary. The ANOVA used the independent factor “condition” with five levels (see Table 1) and the dependent variable “classification accuracy”. If the ANOVA indicated significant differences, we conducted a Newman–Keuls post hoc test to analyze the differences in more detail.

2.5. Analysis 2: improved SSVEP feature extraction in the hybrid runs using harmonics and harmonic phase coupling

The second analysis extended the previous investigation to the hybrid runs. We explored whether an improved SSVEP feature extraction also has a beneficial effect on classification accuracy in the hybrid condition. Here, we compared the results from our previous paper to band power features with three harmonics and HPC. We calculated the band power method with the same three smoothing windows also used before. In short, we used HSD1, HSD2, HSD3, and HPC as listed in Table 1.

We also used logarithmic band power features to quantify ERD over electrodes C3 and C4. We extracted two standard bands (10–12 Hz and 16–24 Hz) for all subjects. In contrast to our previous paper, we excluded the central site Cz, because the main activity for left and right hand motor imagery can be found on contralateral sites C3 and C4 (Pfurtscheller and Berghold, 1989; Pfurtscheller and Lopes da Silva, 1999). Furthermore, since a BCI should function with the fewest possible electrodes, we repeated many analyses in this study with Cz included. However, we found that including site Cz did not substantially improve performance. As with our SSVEP analyses above, we used three different smoothing windows. Hereafter, the three different ERD feature types will be referred to as ERD1 (smoothing length 0.5 s), ERD2 (1 s), and ERD3 (1.5 s).

In summary, we compared seven conditions in this analysis by combining the SSVEP methods with the ERD feature extraction procedure to analyze the hybrid condition. These seven conditions are listed in Table 2. Again, we conducted a one-way repeated measures ANOVA with the independent factor “condition” and the dependent variable “classification accuracy”.

Table 2.

The seven conditions in the second analysis. The SFFS condition is identical to the analysis in our initial paper.

Method	ERD method	SSVEP method
H1	ERD1	HSD1
H2	ERD2	HSD2
H3	ERD3	HSD3
H4	ERD1	HPC
H5	ERD2	HPC
H6	ERD3	HPC
SFFS	SFFS	SFFS

2.6. Analysis 3: true versus artificial hybrid condition

In the third analysis, we compared the hybrid condition with an artificial hybrid condition. We developed the artificial hybrid condition by combining ERD features extracted from the ERD-only runs and SSVEP features extracted from the SSVEP-only runs, where subjects did not perform two tasks simultaneously. In this analysis, we used ERD1, ERD2, and ERD3 to extract ERD features and HSD1, HSD2, and HSD3 to extract SSVEP features with three harmonics and combined them accordingly to form three artificial conditions (with three different smoothing windows). We compared this artificial hybrid condition to the real hybrid conditions H1, H2, and H3 as defined in Table 2. That is, we used the ERD features with three smoothing windows in combination with SSVEP features using three harmonics. In total, we analyzed six different conditions with a one-way repeated measures ANOVA. As usual, the independent factor was “condition” and the dependent variable was “classification accuracy”.

We also used amore fine-grained running classifier in this analysis. Specifically, we split up a trial into segments with 125 ms length and applied a running classifier to estimate the separability of the data of the different conditions.

2.7. Analysis 4: ERD-only and hybrid runs using only ERD features

The fourth analysis assessed whether performing two tasks simultaneously affects motor imagery activity measured with ERD. We compared the performance in the ERD condition analyzed with ERD features ERD1, ERD2, and ERD3 to the performance in the hybrid condition with the same ERD features. As before, those three ERD features correspond to logarithmic band power features at electrodes C3 and C4 using two frequency bands 10–12 Hz and 16–24 Hz, combined with a smoothing window with lengths 0.5 s, 1.0 s or 1.5 s. In summary, we compared six different conditions with a one-way repeated measures ANOVA.

We also used grand average ERDS maps of the ERD-only and hybrid condition in this analysis. The maps were calculated with 2 Hz bands from 5–40 Hz in steps of 1 Hz (plotted along the y-axis) and within the time segment of 0–8 s in steps of 0.05 s (plotted along the x-axis). As a first step, we subtracted the evoked components from the EEG signals for each condition and subject (Kalcher and Pfurtscheller, 1995). Next, the signals were bandpass-filtered, squared and smoothed with a moving average filter (Graimann et al., 2002). Finally, the resulting power values were normalized to the mean power in a reference period between 0.5–1.5 s. The maps are scaled from − 100% (red, ERD) to + 150% (blue, ERS, event-related synchronization). By averaging over all subjects in this study, we obtained grand average ERDS maps.

2.8. Analysis 5: SSVEP-only and hybrid runs using only SSVEP features

Analogous to the previous investigation, this analysis assessed whether performing two tasks at the same time affects visual attention, based on SSVEP activity. Accordingly, we compared the performance in the SSVEP condition analyzed with SSVEP features HSD1, HSD2, HSD3 (band power with three harmonics), and HPC (see also Table 1) to the performance in the hybrid condition with the same SSVEP features. In total, we compared eight different conditions with a one-way repeated measures ANOVA.

2.9. Analysis 6: comparison of ERD-only, SSVEP-only, and hybrid conditions

Analysis 6 investigated whether the hybrid condition yielded better results than the ERD and SSVEP conditions. We analyzed the ERD condition with the three ERD feature types ERD1, ERD2, and ERD3 (corresponding to the three different smoothing windows). Similarly, we extracted bandpower features with three harmonics (HSD1, HSD2, HSD3)as well as HPC features for the SSVEP condition. Finally, we combined those ERD and SSVEP features to analyze the hybrid runs, yielding six different hybrid conditions in total (see also Table 2). In summary, we compared 13 different conditions (ERD1, ERD2, ERD3, HSD1, HSD2, HSD3, HPC, H1, H2, H3, H4, H5, H6) with a one-way repeated measures ANOVA.

2.10. Analysis 7: ERD-only runs using only ERD features and both ERD and SSVEP features

Analysis 7 explored whether a potential improvement in the hybrid condition could be due to the increased size of the feature pool (both ERD and SSVEP features are available to the classifier in this condition). In other words, we questioned if a potential performance improvement could be spuriously caused by the larger pool of different features and feature types. On the other hand, we hypothesized that if an improved performance was actually due to the subjects performing two tasks simultaneously, we should not see higher classification accuracies in the ERD and SSVEP runs using both ERD and SSVEP features. To this end, we analyzed the ERD-only condition with ERD features only and both ERD and SSVEP features (band power with three harmonics) as in the hybrid condition. In summary, we compared six conditions ERD1, ERD2, ERD3, H1, H2, and H3 (see Table 2) with a one-way repeated measures ANOVA.

2.11. Analysis 8: SSVEP-only runs using only SSVEP features and both ERD and SSVEP features

The last analysis is similar to the previous one, but it used data from the SSVEP-only runs instead. We compared the performance when using only SSVEP features HSD1, HSD2, and HSD3 (see Table 1) to using both ERD and SSVEP features H1, H2, and H3 (see Table 2). For statistical evaluation, we once again employed a one-way repeated measures ANOVA.

3. Results

3.1. Analysis 1: improved SSVEP feature extraction in the SSVEP-only runs using harmonics and harmonic phase coupling

The cross-validated maxima of the classification accuracy for all methods are listed in Table 3. In general, all new methods yielded improved results. Subjects S4, S9, S10, S11, S13, and S14 showed the greatest improvement. HSD3, based on band power features with a smoothing window of 1.5 s, was the best method for seven out of 14 subjects, whereas HPC performed best in five subjects.

Table 3.

Cross-validated maxima of the classification accuracy (in %) for all five methods. HSD1, HSD2, and HSD3 refer to band power features with three harmonics and smoothing window lengths 0.5 s, 1.0 s, and 1.5 s, respectively. HPC is the harmonic phase coupling approach, whereas SFFS denotes the results from our previous paper. The last two rows show the mean and standard deviation (STD). The best method for each subject is marked boldface.

HSD1	HSD1	HSD2	HSD3	HPC	SFFS
S1	59.2	58.4	59.9	66.4	65.9
S2	89.4	97.0	94.0	94.7	89.4
S3	65.8	71.5	74.1	66.6	69.1
S4	86.8	88.9	95.3	88.0	77.4
S5	67.9	69.2	71.7	72.7	67.8
S6	67.0	65.5	69.9	68.1	69.1
S7	100.0	100.0	100.0	100.0	100.0
S8	71.5	74.3	75.1	80.0	73.6
S9	87.7	88.7	92.8	91.9	70.6
S10	90.5	92.4	93.3	92.2	84.0
S11	76.4	79.3	84.0	85.3	66.5
S12	90.4	89.5	90.9	86.4	84.5
S13	83.9	84.2	85.3	86.3	72.8
S14	90.2	93.9	98.2	95.0	86.2
Mean	80.5	82.3	84.6	83.8	76.9
STD	12.3	12.8	12.4	11.3	10.3

We found a highly significant difference between the five SSVEP feature extraction methods (F_4,52 = 9.68, Greenhouse–Geisser adjusted p < 0.01). A Newman–Keuls post hoc test revealed that the SSVEP method using SFFS from our first paper was significantly worse than all other improved methods. In addition, HPC and HSD3 (smoothing length 1.5 s) were both significantly better than HSD1 (smoothing length 0.5 s). There was no significant difference between HSD3 and HPC, although the mean of the former method was slightly higher. On the other hand, HPC outperformed HSD3 by over 5% in subjects S1 and S8. Fig. 3(a) summarizes the results of the post hoc tests.

Fig. 3.

(a) Results from the post hoc test in analysis 1. Shaded boxes mark significant differences, and the “<” or “>” signs indicate if the condition on the left was smaller or greater than the condition on top. White boxes indicate non-significant differences. (b) Results from the post hoc test in analysis 4.

3.2. Analysis 2: improved SSVEP feature extraction in the hybrid runs using harmonics and harmonic phase coupling

In contrast to the SSVEP condition, there is only a marginally significant difference between the methods in the hybrid runs (F_6,78 = 3.30, Greenhouse–Geisser adjusted p = 0.060). The means of the seven conditions H1–H6, SFFS (as listed in Table 2) are as follows: 83.1, 85.2, 86.1, 83.4, 84.3, 84.7, and 81.0. Clearly, the old method performs worst, but the difference is not large enough to be significant. Interestingly, subjects S1 and S8 performed best when using the method based on SFFS, whereas the classification accuracy of all other subjects improves when using the new SSVEP feature extraction methods.

3.3. Analysis 3: true versus artificial hybrid condition

There was no significant difference between the true hybrid and artificial hybrid conditions (F_5,65 = 1.64, Greenhouse–Geisser adjusted p = 0.215).

Nevertheless, when comparing the separability of the data with a running classifier, it appears that the artificial hybrid condition resulted in a higher discriminability than the true hybrid condition throughout the whole trial (see Fig. 4). Moreover, the real hybrid condition yielded a slightly higher separability than the SSVEP condition, whereas the ERD data was, in general, poorly separable. The shapes of the true and artificial hybrid classification time courses are almost identical. Therefore, we cannot answer the question if the two task in the true hybrid condition were performed simultaneously or sequentially from this analysis.

Fig. 4.

Time course of classification accuracies obtained by a running classifier in 0.125 ms segments. The plots show results for the three different smoothing windows 0.5 s, 1.0 s, and 1.5 s (from left to right). The three conditions ERD (red, dotted), SSVEP (blue, dashed), and hybrid (magenta, solid) are shown; in addition, the artificial hybrid condition is plotted in green (dash-dotted).

3.4. Analysis 4: ERD-only and hybrid runs using only ERD features

This analysis showed that ERD performance decreases when a visual attention task is added to the motor imagery task. The ANOVA yielded a significant difference of the six conditions (F_5,65 = 4.38, Greenhouse–Geisser adjusted p < 0.05). The performance in the ERD runs was generally higher than in the hybrid runs when using only ERD features. More specifically, a Newman–Keuls post hoc test revealed significant differences for ERD1 in the ERD condition versus ERD1, ERD2, and ERD3 in the hybrid condition, respectively. In addition, ERD2 in the ERD runs yielded a significantly higher accuracy than the same feature type in the hybrid condition (see also Fig. 5(left)). The results of the post hoc test are summarized in Fig. 3(b).

Fig. 5.

Left: Error bars indicating the ERD performance in the ERD-only and hybrid runs (analysis 4). The crosses mark the mean classification accuracy, while the bars denote the 95% confidence intervals around the means. Right: Classification performance in the three conditions ERD-only, SSVEP-only, and hybrid (analysis 6).

The decreased ERD performance in the hybrid condition is also apparent in the ERDS maps of the motor channels C3 and C4 (Fig. 6, first and third row). Clearly, the contralateral ERD in the lower alpha band (around 8–10 Hz) starting shortly after the cue is more lateralized in the ERD condition than in the hybrid condition. We hypothesize that this resulted in a decreased ERD performance in the hybrid runs.

Fig. 6.

Grand average ERDS maps of the three conditions ERD (C3, C4), SSVEP (O1, O2), and hybrid (C3, C4, O1, O2). The vertical line at second 2 denotes the occurrence of the cue. For a detailed description, see main text.

3.5. Analysis 5: SSVEP-only and hybrid runs using only SSVEP features

In contrast to the previous analysis with ERD features, the performance in a visual attention task (SSVEP) did not decrease when a motor imagery task (ERD) was performed simultaneously (F_7,91 = 2.75, Greenhouse–Geisser adjusted p = 0.066). In fact, the means between the SSVEP and hybrid condition are almost identical (82.81 and 82.59, respectively). We confirmed this with a two-way ANOVA with the factors “condition” (two levels “SSVEP” and “hybrid”) and “method” (four levels). While the factor “method” was highly significant (F_3,39 = 8.82, p < 0.01), the more interesting factor “condition” was not significant (F_1,13 = 0.03, p = 0.868), as expected.

These results are corroborated by ERDS maps shown in Fig. 6 (second and fourth row). In contrast to the ERD features (analysis 4), there is no obvious difference between the SSVEP-only and hybrid condition when examining the SSVEP channels O1 and O2. This was reflected in an identical SSVEP performance in both the SSVEP-only and hybrid runs.

3.6. Analysis 6: comparison of ERD-only, SSVEP-only, and hybrid conditions

The comparison between the ERD-only, SSVEP-only, and hybrid conditions resulted in a highly significant difference (F_12,156 = 11.64, Greenhouse–Geisser adjusted p < 0.01). A Newman–Keuls post hoc test revealed that the ERD condition (for all three feature types, ERD1, ERD2, and ERD3) is significantly worse than both the SSVEP and the hybrid condition (see Fig. 5(right)). There was no significant difference between the hybrid and SSVEP conditions. In nine of 14 subjects, our classifier could most accurately distinguish left versus right tasks in the hybrid condition. Our classifier was most accurate for four subjects in the SSVEP condition, and one subject in the ERD condition.

The hybrid condition had the fewest illiterates, followed by the SSVEP and then the ERD conditions. By definition, an illiterate subject cannot effectively control a BCI, meaning that the classification accuracy is below a certain threshold. We used a threshold of 70% in our offline simulation, a value often used in the literature (Perelmouter and Birbaumer, 2000; Kübler and Birbaumer, 2008). Specifically, there were 11 illiterates in the ERD condition, 3 in the SSVEP condition, and 1 in the hybrid condition (averaged over the different methods used within a condition). We conducted a non-parametric Cochran test to assess the differences between the number of illiterates in the three conditions. This test was highly significant (Q = 15.27, p < 0.01). Subsequently, we used a Bonferroni-corrected non-parametric McNemar test to assess pairwise differences between the conditions. The number of illiterates was significantly lower (p < 0.01) in the hybrid condition than in the ERD condition. We did not find statistically significant differences between hybrid and SSVEP as well as ERD and SSVEP conditions, respectively.

3.7. Analysis 7: ERD-only runs using only ERD features and both ERD and SSVEP features

This analysis compared the ERD condition with either ERD features only or both ERD and SSVEP features. The ANOVA did not yield a significant difference between the conditions (F_5,65 = 1.00, p = 0.426). This implies that a greater number of features (and also feature types) could not improve classification accuracy in the ERD condition.

3.8. Analysis 8: SSVEP-only runs using only SSVEP features and both ERD and SSVEP features

Similar to the previous investigation, this analysis compared the SSVEP condition when using either SSVEP features only or both ERD and SSVEP features. Surprisingly, the ANOVA resulted in a highly significant difference (F_5,65 = 8.08, Greenhouse–Geisser adjusted p < 0.01). However, the Newman–Keuls post hoc test revealed significant differences between the pairs HSD3–HSD1, HSD3–HSD2, H1–HSD3, H2–HSD3, H3–HSD1, and H3–H1. From these results, we could not conclude whether there actually was a difference between using only SSVEP features versus using both SSVEP and ERD features. Therefore, we conducted another ANOVA with two factors (first factor “features” with two levels “SSVEP” and “SSVEP and ERD”, second factor “smoothing window” with three levels 0.5 s, 1.0 s, and 1.5 s). This time, the factor that was most interesting for this analysis (“features”) was not significant (F_1,13 = 1.36, p = 0.265), which implies that using both feature types and/or more features did not improve performance. The factor “smoothing window” was highly significant (F_2,26 = 13.48, p < 0.01), whereas the interaction between the two factors turned out to be not significant (F_2,26 = 0.31, p = 0.738).

4. Discussion

In summary, the results lead to four conclusions. First, the newer techniques in this study could improve classification accuracy. Second, the addition of a secondary task produced no significant effect on SSVEP activity, and led to less discriminable ERD patterns. However, our third conclusion is that, despite this latter result, this ERD impairment was overcome by the benefit of adding SSVEP activity that could improve classification. Our fourth conclusion is that the hybrid BCI approach might reduce illiteracy, although there was no significant difference in classification accuracy between the SSVEP and hybrid condition.

We could significantly improve accuracy in the SSVEP condition by using more advanced techniques. The harmonic phase coupling approach (HPC) and the band power method with three harmonics and a smoothing window of 1.5 s performed particularly well. The improved SSVEP methods could also improve classification accuracy in the hybrid runs, although this improvement was only marginally significant.

The hybrid condition did not yield significantly higher classification accuracies than the SSVEP condition. However, the classifier yielded the highest classification accuracies in the hybrid condition for nine out of 14 subjects. This underscores an important benefit of the hybrid approach, which is also apparent from other analyses. Our study, like most BCI studies, revealed significant inter-subject differences. Some subjects might not benefit from the hybrid approach. This is consistent with results from our initial study, in which one subject (S7) attained 100% accuracy in the SSVEP-only condition, and thus gained no improvement from adding an imagined movement task and resulting ERD activity. Although this analysis showed that the hybrid condition produced no significant improvement over all subjects, it also yielded a considerable benefit for some subjects. Therefore, although a hybrid BCI based on our current dual task approach might not help some subjects, it could substantially help other subjects. It could effectively reduce the number of illiterates from 11 in the ERD condition and 3 in the SSVEP condition to 1 in the hybrid condition (averaged over the different methods).

The hybrid condition described here was essentially a dual task paradigm, which leads to two questions. Does the addition of a second task distract subjects, yielding less discriminable ERD and/or SSVEP activity? Second, if the classification accuracy resulting from one of the two brain signals (ERD or SSVEP) declines, would overall classification accuracy necessarily decline as well? Analyses 4 and 5 addressed the first question by showing that ERD activity was less apparent when an SSVEP task was added, but that SSVEP activity was not significantly affected by a secondary ERD task. The third analysis addressed the second question. An artificial hybrid condition, which combined ERD-only and SSVEP-only conditions, showed no significant difference from the true hybrid condition, in which subjects performed two tasks. The second and sixth analyses further suggested that the hybrid condition was comparable to or better than either of the other two conditions. Therefore, even though the dual task approach diminished ERD activity, the supporting information available from a secondary brain signal (at least) made up for any resulting classification reduction.

Distraction is a concern with any BCI system, especially in real-world settings. BCI users must perform intentional mental tasks to accomplish goals. If the user is distracted by external events or other tasks – including tasks required to generate additional brain signals in a hybrid system – then BCI performance may decline. SSVEP activity is reduced when subjects must divide activity among multiple tasks or different regions (Müller et al., 2003; Toffanin et al., 2009). Similarly, work involving event-related potential (ERP) measures of visual attention found that the amplitude of the P300 and other ERPs may diminish when subjects must perform a secondary task, including a visual or motor task (Isreal et al., 1980; Wickens et al., 1983; Sirevaag et al., 1989; Kok, 2001; Matthews et al., 2006, 2009).

However, our study found that simultaneous performance of an imagined movement task did not impair classification of SSVEP activity. This apparent inconsistency with prior literature may stem from the low “distraction quotient” of the secondary task in this study. The dual task studies mentioned above, and other work, typically note that the destructive interference introduced by a secondary task may be reduced if the secondary task involves a different modality or processing resources, and/or seems easy or automatic to the user. Hybrid BCIs should draw on existing principles of dual task integration and strive to minimize destructive interference between the different tasks used to convey information. For example, a P300 BCI that required users to imagine pressing a button instead of counting might be difficult to combine with an ERD BCI based on imagined hand movement. However, a P300 BCI based on imagined counting might be difficult to combine with a BCI that requires users to perform a math task, such as inMillán et al. (2004). Had we instructed subjects to imagine movement observation (third-person imagery) instead of movement execution (first-person imagery) (Neuper et al., 2005), they might have had more trouble focusing visual attention on the LEDs, and thus exhibited worse SSVEP performance in the hybrid condition. Subjects’ dual task performance, and the separability of their associated EEG patterns, might improve with training.

Finally, the seventh and eighth analyses showed that the improvement in the hybrid condition did not result from additional features and/or feature types. This result was not surprising. The SSVEP BCI literature has confirmed results from earlier SSVEP research: SSVEP activity is strongest over occipital sites at the fundamental frequency of the oscillating stimuli and its harmonics (Müller-Putz et al., 2005, 2008; Allison et al., 2008). Therefore, we hypothesized that adding features from central sites at different frequencies would not improve performance. Similarly, the ERD BCI literature, like earlier ERD literature, showed that ERD activity resulting from imagined hand movement is strongest around 9–13 Hz in central sites (Pfurtscheller et al., 1996; Pfurtscheller and Lopes da Silva, 1999).

The tasks described here could lead to a hybrid BCI system that combines ERD and SSVEP BCIs in a particular fashion to improve accuracy. Many other task combinations and potential hybrid BCIs are possible. Different tasks might increase the number of signals or dimensions available, and/or reduce the time per selection, instead of increasing accuracy. When subjects perform two tasks, the EEG activity elicited by either task might be weaker, stronger, or about the same as the EEG activity that either task would have produced in isolation. Novel task combinations might even yield constructive interference, such as by helping subjects to focus on each task. For example, if the left LED were replaced with an oscillating image of a hand grasping, then this visual image might help people focus more strongly on left hand motor imagery. Furthermore, the grasping hand could change with the user’s brain activity, thus providing a realtime feedback mechanism.

Hybrid BCIs should minimize the “distraction quotient” of the additional task by avoiding task combinations that confuse subjects, overburden a particular sensory modality or mental process, or create conflicts between automatic and controlled tasks (Stroop, 1935; Shiffrin and Schneider, 1984). Subjects might perform two tasks sequentially rather than simultaneously (Pfurtscheller, 2009; Pfurtscheller et al., 2010), which could substantially reduce dual task interference. Indeed, it is impossible to rule out the possibility that subjects were rapidly switching between ERD and SSVEP tasks in our current paradigm, and this possibility also merits further study.

In conclusion, a hybrid BCI based on our current protocol and analysis methods might improve accuracy and/or reduce the number of illiterate subjects relative to a comparable simple BCI based on only one task and associated brain signal. This improvement would likely vary across subjects. Hence, some subjects could benefit considerably from this hybrid approach, possibly enough to attain proficiency even if their accuracy with a comparable simple BCI is not adequate for effective communication. This hypothesis should be explored with a BCI (that is, an online system with realtime feedback). We hope that our work also encourages research with other hybrid BCI approaches.