A multi-modal brain–computer interface based on threshold discrimination and its application in wheelchair control

Abstract

In this study, we propose a novel multi-modal brain–computer interface (BCI) system based on the threshold discrimination, which is proposed for the first time to distinguish between SSVEP and MI potentials. The system combines these two heterogeneous signals to increase the number of control commands and improve the performance of asynchronous control of external devices. In this research, an electric wheelchair is controlled as an example. The user can continuously control the wheelchair to turn left/right through motion imagination (MI) by imagining left/right-hand movement and generate another 6 commands for the wheelchair control by focusing on the SSVEP stimulation panel. Ten subjects participated in a MI training session and eight of them completed a mobile obstacle-avoidance experiment in a complex environment requesting high control accuracy for successful manipulation. Comparing with the single-modal BCI-controlled wheelchair system, the results demonstrate that the proposed multi-modal method is effective by providing more satisfactory control accuracy, and show the potential of BCI-controlled systems to be applied in complex daily tasks.

Introduction

Brain–computer interface (BCI) directly transmits the instructions from the human brain to the designated machine terminals by recognizing electrical activities of the nervous system. This technology provides human with a communication channel without physical touch with the controlled objects, which is more efficient and convenient for assistive systems, and shows great innovative significance as well as application value in the field of human–robot communication. BCI has been widely used in various fields such as rehabilitation, disaster relief, entertainment experience, and improving the quality of life for disabled people (Fouad et al. 2015; Mak and Wolpaw 2009; Lebedev and Nicolelis 2006).

In the field of rehabilitation engineering, Electroencephalogram (EEG)-based BCIs for wheelchair control have attracted great attention because of their convenience, non-invasion and low cost (Rebsamen et al. 2010; Wang et al. 2014; Kim et al. 2018; Li et al. 2013). Common modalities used in EEG-based BCIs include steady-state visual evoked potentials (SSVEP) (Cheng et al. 2002; Franois et al. 2010), event-related potentials (ERPs) (Fazel-Reza et al. 2012; Jin et al. 2017), and event-related desynchronization/synchronization (ERD/ERS) potentials produced by motor imagery (MI) (Lafleur et al. 2013; Pfurtscheller and Pfurtscheller 2001). Several purely MI-based BCIs and SSVEP-based BCIs have been developed to realize basic control of external devices (Dong et al. 2018, 2017; Gao et al. 2019). Although the SSVEP-based BCIs has a higher information transfer rate (ITR) and can provide more classification commands, it can only provide discrete commands. MI-based BCIs can generate nearly continuous output in real-time but suffer from the limited number of distinguishable MI tasks (Ma et al. 2017; Ko et al. 2019).

An asynchronous SSVEP-BCI control a wheelchair (Zhang et al. 2020) was proposed. Eight healthy subjects controlled the wheelchair online to complete simple tasks, the average control accuracy was 93%, with possible further improvements to allow the continuous turning angles between 0° and 90°.

To overcome the limitation of using single-modal paradigms, many works have been reported in recent years, such as the multi-modal BCIs (Xu et al. 2021, 2020; Han et al. 2020; Zuo et al. 2020; Yan and Xu 2020; Lamti et al. 2019; Lee et al. 2018). The combination of BCIs for control of an application has been presented as a “multi-modal BCI”. In their paper, the authors described different ways on how a hybrid BCI can be constructed. A hybrid BCI is composed essentially of two or more BCIs that are operated sequentially or simultaneously. Multi-modal BCIs combine different modal EEG and detect at least two brain patterns simultaneously or sequentially to generate control commands (Li et al. 2013; Cao et al. 2014; Huang et al. 2019; Pfurtscheller et al. 2010; Horki et al. 2011; Duan et al. 2019). Lately, a multi-modal BCI was proposed that simultaneously combines P300 and SSVEP to improve the performance of asynchronous control (Li et al. 2013). This system detected the control/idle states and further recognized the target by determining whether both P300 and SSVEP occur in the same group of buttons, which was used to produce “go/stop” commands in wheelchair control (Li et al. 2013). Huang et al. proposed a multi-modal BCI based on MI-EEG and EOG signals. This system was used to control an integrated system (Huang et al. 2019). MI-based switch was used to turn ON/OFF an SSVEP-based BCI (Pfurtscheller et al. 2010). Multi-modal BCI based on MI and SSVEP signals were developed for control engineering (Horki et al. 2011; Duan et al. 2019). Most of these multi-modal BCIs were not applied to multi-dimensional control. In fact, it is a challenge of BCI systems to control devices that require highly accurate and continuous control.

In this paper, a multi-modal BCI system is proposed based on a method of threshold discrimination. The system can accurately determine the current signal modality so that different EEG modalities can be used to increase the number of control commands and improve the performance of asynchronous control. Users can continuously control the wheelchair to turn left/right by imagining left/right-hand movement and control the wheelchair to start, forward, back, stop, turn on/off lights and whistle, by focusing on the SSVEP stimulation panel. The threshold is set based on data analysis from offline experiments. In the online experiments, the biggest correlation coefficient between the current EEG and the SSVEP reference signals is calculated based on the canonical correlation analysis (CCA) (Lin et al. 2006). When the biggest correlation coefficient is less than the set threshold, MI signal classification is performed, otherwise, it is considered that the user is focusing on the flashing light panel and the SSVEP classification is performed. Ten healthy subjects participated in a MI training session, 8 of which with accuracy over 75% were qualified to complete a mobile obstacle-avoidance experiment using the proposed system. The results demonstrate the feasibility of the proposed method.

Methodology

System components

As shown in Fig. 1, the system is composed of a signal acquisition device, an SSVEP stimulation panel, an EEG signal processing module, a wheelchair control module, and an electric wheelchair. A G.GAMMASYS device (g.tec medical engineering, Inc., Austria) was used to capture scalp surface EEG signals for data acquisition. Each user wears an EEG cap (LT 16) with Ag–AgCl electrodes, as shown in Fig. 2. The reference and grounding electrodes are mounted on an ear lobe and the AFz position respectively, as shown in Fig. 2. The EEG signals used for analysis are recorded from 12 electrodes (“Pz,” “PO5,” “PO3,” “POz,” “PO4,” “PO6,” “O1,” “Oz,” “O2,” “C3,” “Cz,” and “C4”), which are marked in red in Fig. 2 (Kuhlman. 1978; Middendorf et al. 2000). The EEG signals are amplified by a g.USBamp RESEARCH amplifier with a sampling rate of 256 Hz. The impedances between the scalp and all electrodes are maintained below 5 kΩ. On the SSVEP stimulation panel, the stimuli are delivered through 6 groups of LEDs flashing at different frequencies (25-30HZ), and the duty cycle is 50%. The algorithm of the EEG signal processing module is performed on the computer. All the computations are carried on a Lenovo computer (Inter (R) Core (TM) i5-4590 CPU 3.30 GHz) with the software MATLAB (2017a). The main control chip of the wheelchair control module is a single-chip microcomputer (STC89C51), combined with Bluetooth module (BT04-A, following the V4.0BLE Bluetooth specification), which provides the electric wheelchair with a communication channel to replace the joystick mechanical control. The signal transmission rate is 1150kbps, and the command transmission delay time can be controlled within 500 ms.

Fig. 1.

Hardware system composition

Fig. 2.

Electrodes names and distribution. The electrodes used for analysis are marked in red

Control strategy

The system provides 8 commands: start, stop, forward, back, turn on/off lights, whistle, turn left, and turn right, as shown in Table 1 These commands are used for constructing the effectively control for the wheelchair.

Table 1.

The control command and their corresponding mental tasks

Control commands	Evoking tasks
Start	Focusing on the button flashing in 25 Hz
Stop	Focusing on the button flashing in 26 Hz
Forward	Focusing on the button flashing in 27 Hz
Back	Focusing on the button flashing in 28 Hz
Turn on/off lights	Focusing on the button flashing in 29 Hz
Whistle	Focusing on the button flashing in 30 Hz
Turn left	Imagine left-hand movement
Turn right	Imagine right-hand movement

Threshold discrimination and classification algorithm

A threshold strategy is used to distinguish different modalities of EEG signals. Then, SSVEP or MI-based signals classification is performed. Figure 3 shows the procedure for the SSVEP and MI detection. The EEG signals are collected in real-time and filtered between 8 and 40 Hz. Then performing canonical correlation analysis (CCA) with the constructed 6 reference signals (25–30 HZ) to obtain the biggest correlation coefficient R.

$$R(X,Y)=\frac{\text{cov}(X,Y)}{\sqrt{D(X)}\sqrt{D(Y)}}$$ 1

where $X$ is multi-channel EEG signals and $Y$ is reference signals. $\text{cov}(X,Y)$ is the covariance of $X$ and $Y$. $D(X)$ and $D(Y)$ are the variances of $X$ and $Y$, respectively. The value of correlation coefficient $R$ is in [− 1, 1]. An absolute value of $R$ closer to 1 indicates a higher linear correlation between $X$ and $Y$. Sinusoidal signals are used as the reference signals ${\mathbf{Y}}_f$:

$${\mathbf{Y}}_f=\left[\begin{array}{c}sin(2\pi ft)\\ cos(2\pi ft)\\ ⋮\\ sin\left(2\pi N_{h}ft\right)\\ cos\left(2\pi N_{h}ft\right)\end{array}\right]$$ 2

where $f$ is the stimulation frequency and $N_h$ is the number of harmonics.

Fig. 3.

A flow chart for the method used in the system

Figure 3 shows the flow chart of the proposed method, where R is the maximal correlation coefficient and I is the threshold. R is compared with the threshold I. If R < I, then the user is considered performing MI, and MI classification is performed. If R ≥ I, the user is considered focusing on the flashing lights, then the SSVEP classification is performed. The threshold will be determined later in this paper.

SSVEP classification

When the current signal is identified as SSVEP, after threshold discrimination, SSVEP classification is performed. Under no training conditions, the filter bank canonical correlation analysis (FBCCA) method enhances the CCA-based frequency detection of SSVEP signals and shows good performance in extracting task-related components, which is therefore used for SSVEP classification in the proposed system. Figure 4 shows the flowchart of FBCCA, consisting of three major procedures: (1) filter bank analysis; (2) performing CCA between SSVEP sub-band components and sinusoidal reference signals; and (3) target identification (Chen et al. 2015). In the first step, a filter bank analysis is performed for sub-band decompositions with multiple filters having different passbands. After the filter bank analysis, the standard CCA process is applied to each of the sub-band components $\left({\mathbf{X}}_{{\text{SB}}_n,},n=1,2,\dots ,N\right)$ separately, to calculate the correlation coefficients between the sub-band components and the predefined reference signals corresponding to all stimulation frequencies. For the kth reference signal $(Y_{f_k},k=1,2,\dots ,6)$, its correlation coefficients with $N$ sub-band components form a 1-by-N correlation vector ${\rho }_k$ defined as follows:

$${\rho }_k=\left[\begin{array}{c}{\rho }_k^1\\ {\rho }_k^2\\ ⋮\\ {\rho }_k^N\end{array}\right]=\left[\begin{array}{c}\rho \left(X_{S{B}_1}^T{W}_X\left(X_{S{B}_1}{Y}_{f_k}\right),Y^T{W}_Y\left(X_{S{B}_1}{Y}_{f_k}\right)\right)\\ \rho \left(X_{S{B}_2}^T{W}_X\left(X_{S{B}_2}{Y}_{f_k}\right),Y^T{W}_Y\left(X_{S{B}_2}{Y}_{f_k}\right)\right)\\ ⋮\\ \rho \left(X_{\mathit{SBN}}^T{W}_X\left(X_{\mathit{SBN}}{Y}_{f_k}\right),Y^T{W}_Y\left(X_{\mathit{SBN}}{Y}_{f_k}\right)\right)\end{array}\right]$$ 3

where ${\rho }_k$ indicates the correlation coefficient between $X$ and $Y$. In frequency detection of SSVEPs, $X$ indicates multi-channel SSVEPs and $Y$ is reference signals. $x$ and $y$ are their weighted linear combinations respectively. $W_X$ and $W_Y$ are the weight vectors of $X$ and $Y$. Therefore, $x=X^T{W}_X$ and $y=Y^T{W}_Y$.

$$\underset{W_X,W_Y}{max}\rho (x,y)=\frac{E\left[W_X^{T}X{Y}^T{W}_Y\right]}{\sqrt{E\left[W_X^{T}X{X}^T{W}_X\right]E\left[W_Y^{T}Y{Y}^T{W}_Y\right]}}$$ 4

Fig. 4.

Flowchart of the FBCCA method for frequency detection of SSVEPs

A weighted square sum of the correlation coefficients corresponding to all sub-band components, i.e., ${\rho }_k^1,\dots ,{\rho }_k^N$, is defined as the feature for classification:

$${\tilde{\rho }}_k=\sum _{n=1}^{N}w(n)\times {\left({\rho }_k^n\right)}^2$$ 5

where n is the index of the sub-band. According to the fact that the signal-to-noise ratio (SNR) of SSVEP harmonics decreases as the response frequency increases, the weights for the sub-band components are defined as follows:

$$w(n)=n^{-a}+b,n\in [1N]$$ 6

where a and b are constants that maximize the classification performance. Grid search method using an offline analysis is used to determine a, b and N. From experimental data, we get a = 1.75, b = 0.25 and N = 5. Finally, ${\tilde{\rho }}_k$ corresponding to all SSVEP stimulation frequencies $\left(i.e.,{\tilde{\rho }}_1,\dots ,{\tilde{\rho }}_6\right)$ is used for determining the frequency of the flashing light that the user focuses on, indicated by the maximal in ${\tilde{\rho }}_k$.

MI signals classification

When the current signal is identified as MI signal after threshold discrimination, the MI classification is performed. As shown in Fig. 5, the MI signal classification is divided into the offline model training and the online classification processes. In the offline model training process, each user is required to complete several tasks of imagining left/right-hand movements. For the feature extraction, the discriminative filter bank common spatial pattern (DFBCSP) method (Thomas et al. 2009) is applied. The flow chart of DFBCSP is shown in Fig. 5.

Fig. 5.

Flowchart of the DFBCSP method for MI

The DFBCSP selects the appropriate FIR filter coefficients and divides the original frequency band into multiple sub-bands. If every M th coefficient of a finite impulse response filter $h(n)$ is kept unchanged and all other coefficients are replaced by zeros, we get $h^{\prime }(n)$:

$$h^{\prime }(n)=h(n)·c_M(n)$$ 7

where

$$c_M(n)=\left\{\begin{array}{c}1;\text{for}n=kM,k=0,1,2,\dots M-1\hfill \\ 0;\text{otherwise}\hfill \end{array}\right)$$ 8

Find the power of each sub-band:

$$P\left(f_i,t\right)=\frac{1}{T}\sum _{n=1}^T{x}_{t,f}^2(n)$$ 9

Calculate the Fisher Ratio for each frequency band, and select the 4 frequency bands with the highest FR score:

$$F_R(f)=\frac{S_B}{S_W}$$ 10

where

$$S_W=\sum _{k=1}^C\sum _{t=1}^{n_k}{\left(P_t-m_k\right)}^2$$ 11

and

$$S_B=\sum _{k=1}^C{n}_k{\left(m-m_k\right)}^2$$ 12

are the within-class variance and between-class variance, respectively, $m$ is the total average, $m_k$ is the average for class $k$, ($k$ = 1, 2), $C$ is the number of classes, and $n_k$ denotes the number of trials for class $k$.

The original data was passed through the filters in the four frequency filters to obtain the CSP features, and the hybrid kernel function relevance vector machine (HKF-RVM) (Dong et al. 2020) is used for classification.

Experiments

Two types of experiments (offline and online) are designed in this paper. Ten healthy subjects (3 female and 7 males, aged between 25 and 27 years) participated in these experiments. Three subjects (A1, A2, A3) have BCI experiences, while the other seven subjects have no prior experience. All experiments are carried out according to the principles expressed in the Declaration of Helsinki. Before participating the experiments, all subjects get a detailed understanding of the purpose and possible consequences of the experiments.

Offline experiments for data collection

MI offline data collection

The workflow is illustrated in Fig. 6. Each subject is seated in a comfortable chair facing a computer screen. At t = 0 s, the experiment starts with a fixation cross appears on the screen, and a short beep informs the subject the start of experiment; at t = 2 s, a directional arrow appears on the screen (left, right, up or downwards) and remains on the screen for 1.25 s, at the same time, the subject is reminded to start preparing for a motor imagery task according to the direction arrow. The EEG data is recorded from t = 3 s and the motor imagery task continues until t = 6 s.

Fig. 6.

The paradigm of MI training data collection

The next session is performed after a short interval. The data were used to train HKF-RVM models offline.

SSVEP offline data collection

The workflow is illustrated in Fig. 7. The subject is requested to relax, and then start the SSVEP stimulation induction program. The stimulations of flicking targets with predefined frequency are repeated with break periods (no flickers) between two periods. The duration of each stimulation is 7 s separated by 4 s rest.

Fig. 7.

The paradigm of SSVEP data collection

Before each stimulation, one of the LED will be pointed by a yellow arrow. The subject must focus on the marked LED for the whole stimulation period. Throughout the experiment, each LED will be focused six times in sequence. At the end of the process, a file containing annotated EEG signals will be generated for subsequent data analysis.

Online mobile obstacle avoidance experiment

The purpose of this experiment is to evaluate the wheelchair manipulating performance of participants and the feasibility of the proposed method. A challenging task is designed, where subjects are asked to drive a wheelchair to follow a predefined trajectory (about 45 m) containing a narrow straightway, a door entrance, facing to obstacles, rotating around obstacles, and turning around obstacles with no collision. The layout of the experiment and tasks are shown in Fig. 8.

Fig. 8.

Wheelchair control experiment with difficult tasks

To assess the subject’s feeling of control, after the online experiment the subjects filled out a questionnaire with three questions: (1) rate your ability to control the wheelchair to walk straight by focusing on the flickering lights; (2) rate your ability to control the turning of the wheelchair by motor imagery; (3) rate the control capability of the system compared with a single MI-BCI and a single SSVEP-BCI. The answer to each of the questions is a scale between 1 (low) and 10 (high).

Results

MI training results

In this study, each of the 10 subjects completed 8 MI training sessions. According to the binomial test theory, under normal circumstances, if the experimental significance reaches the 0.05 level, then it is considered that significant differences exist between the data. The corresponding p value of the experiment is calculated in Eq. (13), which should be less than 0.00625 (0.05/8) for the 8 sessions.

$$p=1-\sum _{i=0}^m\left(\begin{array}{c}n\\ i\end{array}\right)0.5^n{I}_{(0,1,\dots ,n)}(i)$$ 13

In Eq. (13), n is the number of trials in a session (20 in this case), and m is the number of the correctly predicted trials in a session. The indicator function $I_{(0,1,\dots ,n)}(i)$ ensures that m only adopts values of $0,1,...,n$. By this formula, to ensure that p is less than 0.00625, m should be greater than 14. Therefore, the minimum number of correct trials required in a session is 15 (accuracy rate is about 75%). We keep only the subjects whose MI classification accuracy is higher than 75% to participate in further experiments.

According to the experimental results, 8 of the 10 subjects achieved the MI classification accuracy above 75%, as shown in Table 2 Among them, the three subjects with MI experience (A1, A2 and A3) had the highest accuracy rates of 95%, 100% and 95% respectively. The average MI accuracy and standard deviation of these eight subjects are shown as blue bars in Fig. 9.

Table 2.

The best results of the eight subjects in the MI-/SSVEP-based sessions

Subjects	Gender	Age	MI accuracy (%)	SSVEP accuracy (%)
A1	Male	25	95%	97.2%
A2	Male	24	100%	100%
A3	Female	23	95%	100%
A4	Male	23	90%	94.4%
A5	Female	23	85%	97.2%
A6	Male	24	90%	94.4%
A7	Male	23	80%	97.2%
A8	Male	24	85%	94.4%
Mean ± SD	–	–	90% ± 0.06	96.85% ± 0.02

Fig. 9.

The average accuracies and standard deviations of the eight selected subjects in the MI/SSVEP sessions

SSVEP offline data analysis

To verify the reliability of the collected SSVEP signals, data epochs are extracted to SSVEP EEG data. Epochs of each subject are extracted. The Fast Fourier Transform (FFT) method is used to perform spectrum analysis. The frequency response curves of the EEG signal (using subject A3 as an example) under the six stimuli are shown in Fig. 10. The EEG signals have significant peak at the stimulation frequencies.

Fig. 10.

Frequency response curve of the EEG signals for subjects at different stimulus frequencies. a 25 HZ, b 26 HZ, c 27 HZ, d 28 HZ, e 29 HZ, f 30 HZ

Threshold setting

To obtain the threshold for determining the existence of SSVEP, the CCA is applied to the MI and SSVEP data with the constructed 6 types of SSVEP reference signals, respectively. For MI data, each subject performs 3 trials, and each trial gets 10 comparisons. For SSVEP data, each subject performs 3 trials, and each trial has 6 comparisons. The average and standard deviation of the biggest correlation coefficient between MI and SSVEP data and the reference signal for each subject are shown in Fig. 11. Most of the biggest correlation coefficient between the MI data and reference signals are lower than 0.5, except subject A6 got 0.52 for one time. However, for SSVEP and reference signals, most of the biggest correlation coefficient are higher than 0.5, except subject A7 got 0.45 for one time. Therefore, the biggest correlation coefficient threshold of 0.5 is selected to determine if a subject is focusing on SSVEP stimuli or not.

Fig. 11.

The average and standard deviation of MI and SSVEP the biggest correlation coefficient for each subject

Motorized wheelchair control experiment results

To validate the proposed system in real applications, it is applied to control a motorized wheelchair. The wheelchair control is divided into the following tasks as shown in Fig. 8.

• Task 1: proceeding straight ahead in the corridor;

• Task 2: passing through the entrance after a left turning;

• Task 3: avoiding obstacles (chairs) in the room;

• Task 4: rotating around obstacles (a chair and a table).

All subjects completed all the four tasks after some trials, with some collisions occurred for some subjects. The experimental results (‘O’ and ‘X’ to represent ‘Success’ and ‘Fail’, and the number means the number of collisions) and ITR for each subject in this experiment are illustrated in Table 3. Subjects A1 and A2 successfully completed all tasks without any collisions. A3, A6, and A8 had a scratch during either crossing the entrance or avoiding obstacles. A4 successfully completed the first two tasks, but scratched when avoiding obstacles and rotating around obstacles. A7 completed tasks 1 and 4, but a steering error occurred during obstacle avoidance and a collision occurred, which is relevant to the mechanical flaws of the wheelchair steering control system.

Table 3.

The collisions of the real wheelchair control experiment

Subjects	Task 1	Task 2	Task 3	Task 4	ITR (bits/min)
A1	O	O	O	O	230
A2	O	O	O	O	210
A3	O	X 1	O	O	200
A4	O	O	X 1	X 1	175
A5	O	X 1	O	X 1	130
A6	O	O	X 1	O	160
A7	O	O	X 2	O	140
A8	O	O	X 1	O	175
Mean	–	–	–	–	177.5

Subjective measures

The subjects rated their ability to control the wheelchair to go straight with 9.3 ± 0.7 (on a scale from 1, low, to 10, high). To turning control of the wheelchair was rated 7.5 ± 1.2. Finally, the overall control ability compared with MI-BCI rating (9.6 ± 0.5) is higher than SSVEP-BCI (9 ± 0.5). The MI BCI has higher control ability due to its capacity of continuous control.

Comparison with the single modal BCIs

In our previous work, we investigated brain-controlled wheelchair system based on SSVEP (Zhang et al. 2020). Applying this system to the online brain-controlled wheelchair experiment in this paper, each subject successfully completed the Task 1. In the Task3 and Task4 (when the turn is continuously controlled), although the recognition accuracy of SSVEP was very high, the task failed because the subjects could not control the turning angle well. In the later stage of the experiment, the classification accuracy decreased due to visual fatigue. The online experiments achieved an average accuracy of 70.94% with an ITR of 130 bit/min.

Discussion and conclusion

A hybrid BCI system is composed essentially of two or more BCIs that are operated sequentially or simultaneously. The combination of at least one BCI and other assistive technologies also constitutes a hybrid BCI. In this work, we proposed a threshold discrimination method to distinguish between SSVEP and MI signals, to form a multi-modal BCI. Two BCIs are operated for controlling a wheelchair. The user could continuously control the wheelchair to turn left/right by MI-based BCI and the SSVEP-based BCI provides another 6 commands for the wheelchair control. The hybrid design presented here allows users to operate both BCIs continuously with two different purposes serving the common goal of wheelchair control. The method increased the number of control commands and improved control performance. In our experiments, most subjects were able to control a wheelchair by means of the proposed multi-modal BCI to complete challenging tasks designed based on real living environment. Though there was time variation among subjects, most subjects successfully completed the tasks after some trials. These tasks are not easy to complete even using a joystick.

According to the evaluation of the subjects after use, it was proved that the BCI system is robust and all subjects were well adapted to the system.

When the information transmission rate (ITR) is concerned, based on the time used in the experiment and the number of control commands issued by the system, the system outputs a control command every 1.5 s. Compared with the single-mode BCI system, this control system has obvious advantages. Therefore, the proposed method has a higher efficiency.

For the SSVEP, the stimulation frequency can be selected between 1 and 100 Hz, while significant spectrum peaks were found for the frequency in 4–30 Hz. The MI relevant frequencies are relatively low (for instance, the μ rhythm is in 8–13 Hz and β rhythm in 13–30 Hz). Therefore, the stimulation frequency selected by the system is in 25–30 HZ for clear peaks in SSVEP signal spectrum power and small overlap with MI signal frequencies.

Data analysis showed that in 240 experiments with 8 subjects, only one trial got the biggest correlation coefficients between the MI signal and the reference signal bigger than 0.5 (0.52) during SSVEP stimuli, and all the rest were well lower than 0.5 (below 0.45). Therefore, the selected threshold 0.5 is feasible.

The proposed method combined SSVEP and MI signals to increase the number of control commands and improve the performance of BCI, by an exclusive modal selection strategy. MI provided two separable tasks, and SSVEP provides six. If one task generates one control command, then a total of 8 control commands can be obtained. If one control command is generated on the completion of every two tasks, the control commands can be expanded to 24 by a simple coding strategy. It will be interesting and valuable to combine the detection of different modal signals through the threshold discrimination to further extend the control command set, which will be our work in the future.

Data availability statements

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.