Review of brain encoding and decoding mechanisms for EEG-based brain–computer interface

Abstract

A brain–computer interface (BCI) can connect humans and machines directly and has achieved successful applications in the past few decades. Many new BCI paradigms and algorithms have been developed in recent years. Therefore, it is necessary to review new progress in BCIs. This paper summarizes progress for EEG-based BCIs from the perspective of encoding paradigms and decoding algorithms, which are two key elements of BCI systems. Encoding paradigms are grouped by their underlying neural meachanisms, namely sensory- and motor-related, vision-related, cognition-related and hybrid paradigms. Decoding algorithms are reviewed in four categories, namely decomposition algorithms, Riemannian geometry, deep learning and transfer learning. This review will provide a comprehensive overview of both modern primary paradigms and algorithms, making it helpful for those who are developing BCI systems.

Introduction

A brain–computer interface is a new method for the human or animal brain to interact with the external world without normal neural pathways (Wolpaw and Wolpaw 2012). BCIs are capable of translating measured brain signals to controlling commands for operating computerized devices, such as prosthetic arms. A typical BCI system requires neural signal acquisition, paradigm design, decoding algorithms and feedback. Neural signal acquisition technologies are basic tools to measure brain activities. Paradigms are the designed mental tasks for modulating neural signals. Decoding algorithms are responsible for translating the measured neural signals to commands and the feedback to the user.

One of the neural signal acquisition technologies for BCIs is electroencephalography (EEG), discovered by Berger (1929). Comparing to other neuroimaging methods, EEG is the most widely used biosensing technology in BCI studies due to its non-invasive, high temporal resolution and low-cost characteristics. EEG-based BCI systems have shown promise in many applications, such as post-stroke rehabilitation (Silvoni et al. 2011), disease detection (Nakanishi et al. 2017b), emotion recognition (Zheng et al. 2014), quadcopter control (LaFleur et al. 2013) and video games (Kerous et al. 2018).

Paradigm design is crucial to determine the basic type of an EEG-based BCI. Brain activities usually occur concurrently therefore it is difficult to decompose a specific type of activity directly. In order to elicit the desired brain activities, a well-designed paradigm is used during the signal-acquisition phases. That is, the paradigm itself acts as an encoder that modulates the brain to generate the target brain activities. Many BCI paradigms have emerged over the past few decades. Some are designed to elicit motor-related and sensory-related brain activities, e.g. the motor imagery paradigm. Some entrain the visual cortex, e.g. the steady-state visually evoked potential paradigm, and others are related to cognitive brain activities, e.g. the P300 paradigm.

On the other hand, decoding algorithms aim to decode the target brain activity from measured signals. Decoding algorithms for BCIs are similar to machine learning algorithms, but there are also differences between them. Although the paradigm has already encoded the target activity from mixed brain activities, the signal-to-noise ratio (SNR) of measured EEG signals is still too low to decode with simple machine learning methods. Besides, paradigms usually invoke different EEG signal patterns, requiring to design dedicated decoding algorithms for different paradigms. Many efficient decomposition algorithms have been developed to extract features, e.g. common spatial patterns for motor imagery. Recently, Riemannian geometry and deep learning methods are increasingly explored by researchers in the BCI community. Transfer learning algorithms have also been introduced to alleviate the inter-subject variability problem in BCIs.

This review summarizes recent progress for EEG-based BCIs from the perspective of encoding paradigms and decoding algorithms. This paper is organized as follows. “Encoding paradigms” provides a summary of primary encoding paradigms for BCIs and some important paradigms are shown in Table 1. “Decoding algorithms” describes current BCI decoding algorithms and some recent algorithms are compared in Table 2. “Conclusion” concludes the paper.

Table 1.

Summary of primary encoding paradigms for BCIs

Paradigm	References	Contribution
MI	Ramoser et al. (2000)	Left hand and right hand movement imagery
	Obermaier et al. (2001)	Left hand, right hand, feet and tongue movement imagery
	Yi et al. (2013)	Compound limb motor imagery (both hands, left hand combined with right foot and right hand combined with left foot)
	Lee et al. (2020)	Elbow extension, hand grasping and wrist twisting imaginations of the same arm
	Ofner et al. (2017), Chu et al. (2020)	6 motor imaginations of the same limb (elbow flexion/extension, forearm supination/pronation, hand open/close)
	Edelman et al. (2015b)	4 motor imaginations of the right hand (flexion, extension, supination and pronation)
	Wang et al. (2017)	3 force load imaginations of the right hand clenching
MRCP	Gu et al. (2009)	The rebound rate of MRCPs may indicate the speed of movements
	Bradberry et al. (2010, 2011)	The possibility to decode 3D hand velocity from EEG signals
	Kim et al. (2014)	Non-linear methods are robust to eye-movement contamination
	Schwarz et al. (2017)	A paradigm to decode 3 reach-and-grasp actions from EEG signals
	Wang et al. (2020)	The possibility of decoding movement intention before actual movements
SSSEP	Muller-Putz et al. (2006)	Vibratory stimulations are performed on index fingers
	Breitwieser et al. (2012)	Vibrations for five fingers of the right hand
	Su et al. (2020)	3 stimulation intensities on index fingers
SSVEP	Jia et al. (2010)	Joint frequency and phase modulation scheme
	Min et al. (2016)	A paradimg with grid-shaped line array
	Nakanishi et al. (2017a)	A paradigm of 40 SSVEP stimuli
	Tang et al. (2019, 2020b)	A multi-focal SSVEPs (mfSSVEPs) paradigm
miniature aVEPs	Xu et al. (2018)	A BCI speller with miniature asymmetric visual evoked potentials (miniature aVEPs)
SSaVEP	Yue et al. (2020)	A Steady-State asymmetrically Visual Evoked Potential (SSaVEP) paradigm
P300	Farwell and Donchin (1988)	A row column (RC) paradigm
	Guan et al. (2004), Guger et al. (2009)	A single character (SC) paradigm
	Fazel-Rezai and Abhari (2009)	A region-based (RB) paradigm
	Townsend et al. (2010)	A checkerboard paradigm
RSVP	Acqualagna et al. (2010), Acqualagna and Blankertz (2013)	A rapid serial visual presentation (RSVP) paradigm presenting all characters sequentially
	Lin et al. (2018)	Triple-character presentation in RSVP
ErrPs	Ferrez and Millán (2008)	The study detects ErrPs in a simulated human-robot task
	Chavarriaga and Millán (2010)	An automatic cursor’s movement system with ErrPs detection
	Salazar-Gomez et al. (2017)	A robotic control system with ErrPs detection
CVSA	Tonin et al. (2013)	A paradigm of keeping covert attention on 2 different orientations
	Gaume et al. (2019)	The relation of the visual sustained attention level with the task difficulty
	Ahmadi et al. (2020)	A paradigm of detecting CVSA on two orientations with changing luminance
Hybrid	Allison et al. (2010)	A paradigm that executes MI and SSVEP simultaneously
	Pfurtscheller et al. (2010b)	A paradigm that switches MI and SSVEP tasks sequentially
	Long et al. (2012)	A paradigm that control a wheelchair with P300 and MI
	Yao et al. (2013), Yi et al. (2017)	A paradigm that combines SSSEP and MI
	Mousavi et al. (2020)	An online paradigm executing MI tasks with detection of ErrPs
	Panicker et al. (2011)	An asynchronous P300-speller in which SSVEP was used to recognize subjects’ control states
	Yin et al. (2013a)	A 32-character speller with a flickering stimulus presented in the P300 paradigm
	Xu et al. (2020c)	A large-size P300 and SSVEP-B speller with over 100 commands

Table 2.

Summary of recent decoding algorithms for BCIs

Method	References	Paradigm	Dataset	Score		Contribution
				Accuracy (%)	AUC
Decomposition methods	Chang et al. (2019)		In-house(10,)			An automatic artifact removal approach
	Zhang et al. (2018d)	MI	BCI Competition 3 Dataset 3a(3, 4 classes) BCI Competition 4 Dataset 2a(9, 4 classes) BCI Competition 4 Dataset 2b(9, 2 classes)	88 83 84		A joint sparse optimization of filter bands and time windows with temporal smoothness constraint
	Jin et al. (2020)	MI	BCI Competition 3 Dataset 4a(5, 2 classes) BCI Competition 4 Dataset 1(4, 2 classes)	86 70		A fusion framework based on the Dempster-Shafer theory
	Wong et al. (2020a)	SSVEP	Tsinghua Dataset(35, 40 classes)	82		A new learning scheme utilizing multiple stimuli for SSVEP-based BCIs
	Xiao et al. (2019)	aVEP P300 RSVP mVEP	In-house(12, 2 classes) EPFL Dataset(8, 2 classes) In-house(12, 2 classes) BNCI2015010(11, 2 classes) BNCI2015007(11, 2 classes)		0.73 0.77 0.82 0.80 0.73	Discriminative canonical pattern matching (DCPM) for ERP-based BCIs
	Gurve et al. (2020)	MI	In-house(10, 2 classes)	96		Channel selection method with Non-Negative Matrix Factorization (NMF)
Riemannian geometry	Xu et al. (2020a)	MI	BCI Competition 4 Dataset 2a(9, 2 classes) Cho2017(49, 2 classes) MunichMI(10, 2 classes) PhysionetMI(109, 2 classes) Shin2017A(25, 2 classes) Weibo2014(10, 2 classes) Zhou2016(4, 2 classes)	86 73 88 67 66 82 90		A dimension reduction method for Riemannian methods which reduces the time cost of computation
	Chu et al. (2020)	MI	In-house(12, 6 classes)	80		Partial least squares regression with the tangent features
Deep learning	Lawhern et al. (2018), Waytowich et al. (2018)	MI MRCP P300 ErrPs SSVEP	BCI Competition 4 Dataset 2a(9, 4 classes) In-house(13, 2 classes) In-house(15, 2 classes) Kaggle BCI Challenge(26, 2 classes) SCCN CCA Dataset(10, 12 classes)	67 80	0.8 0.92 0.82	EEGNet with depth-wise separable convolution
	Dai et al. (2020)	MI	BCI Competition 4 Dataset 2a(9, 2 classes) BCI Competition 4 Dataset 2b(9, 2 classes)	91 87		A hybrid-scale CNN architecture with a data augmentation method
	Ravi et al. (2020)	SSVEP	In-house(21, 7 classes) SCCN CCA Dataset(10, 12 classes)	92 92		A CNN network with complex spectrum features as inputs
	Xing et al. (2020)	SSVEP	In-house(23, 4 classes)	91		A comparing network architecture based on CNN inspired by template matching
	Ma et al. (2020)	MI	BCI Competition 4 Dataset 2a(9, 2 classes)	96		Band selection and PSD feature extraction as preprocessing steps for inputs
Transfer Learning	Rodrigues et al. (2018)	MI P300 SSVEP	PhysionetMI(109, 2 classes) Cho2017(50, 2 classes) BCI Competition 4 Dataset 2a(9, 4 classes) BNCI2014002(15, 2 classes) BNCI2015001(13, 2 classes) MunichMI(10, 2 classes) In-house(24, 2 classes) SSVEP(12, 3 classes)		0.67 0.66 0.79 0.73 0.65 0.73 0.75 0.82	Riemannian Procrustes Analysis (RPA) with translation, scaling, and rotation transformations
	Zhang and Wu (2020)	MI RSVP ErrPs	BCI Competition 4 Dataset 1(7, 2 classes) BCI Competition 4 Dataset 2a(9, 2 classes) PhysionetRSVP(11, 2 classes) Kaggle BCI Challenge(16, 2 classes)	83 76 68 66		Manifold embedded knowledge transfer (MEKT) inspired by TCA and JDA
	Li et al. (2020)	P300	BNCI2014008(8, 2 classes) In-house(10, 2 classes)		0.83 0.83	Use xDAWN for feature extraction and RIemannian mean method for aligning
	Chiang et al. (2020)	SSVEP	In-house(10, 40 classes)	77		A least-squares transformation (LST)-based transfer learning framework for SSVEP BCIs

Scores in transfer learning were reported in the cross-subject scenario with leave-one-subject-out cross-validation, while others were reported in the within-subject scenario with k-fold cross-validation. The first number in the bracket of datasets is the number of subjects and the second is the number of classes

Encoding paradigms

Sensory- and motor-related paradigms

Sensory- and motor-related paradigms can modulate neural signals originated in the primary somatosensory cortex and primary motor cortex, respectively. Steady-state somatosensory evoked potential (SSSEP) is a sensory-related paradigm. Motor imagery (MI) and movement-related cortical potential (MRCP) are two popular paradigms focusing on motor-related functions.

MI is a paradigm that a subject imagines performing a movement without executing the actual movement or tensing the muscles (Mulder 2007). The MI paradigm can change the sensorimotor rhythms (SMRs) in the primary sensorimotor area (Pfurtscheller and Neuper 1997). Event-related desynchronization (ERD) and event-related synchronization (ERS) are the most common phenomena in SMRs for the MI-based BCIs. The former decreases power in certain frequency bands, usually the alpha (8–13 Hz) and beta (14–26 Hz) bands. In contrast, the latter increases power in certain frequency bands, like the gamma (>30 Hz) band. ERD and ERS are considered to be related to the decrease and increase in synchronization of the corresponding neuronal populations, respectively (Pfurtscheller and Da Silva 1999). The MI-based BCI outputs commands by detecting distinguishable ERD/ERS from the EEG signals.

The early MI paradigms focused on imagining movements with different limbs, which were expanded from 2-class MI tasks (left and right hands) (Ramoser et al. 2000; Pfurtscheller et al. 1997) to 4-class MI tasks (left hand, right hand, tongue, and feet) (Obermaier et al. 2001; Morash et al. 2008). Instead of imagining simple limb movements, Yi et al. designed compound limb and sequential limb MI tasks (Yi et al. 2013, 2016). Ofner et al. proposed a paradigm which requires subjects to imagine 6 MI tasks of the same limb, namely elbow flexion/extension, forearm supination/pronation and hand open/close, although the classification accuracy of MI tasks was slightly above the random level (Ofner et al. 2017). This paradigm has been further studied recently with improved classification methods. Lee et al. asked subjects to imagine elbow extension, hand grasping and wrist twisting actions (Lee et al. 2020). Chu et al. adopted the same Ofner’s paradigm (Chu et al. 2020). Both of them achieved a better classification accuracy that can be applied in practical applications (84% and 80.5%, respectively). Modern MI paradigms concentrate on imagining more precise movements. Edelman et al. designed a MI paradigm for the complex tasks of a single limb, namely 4 hand gestures of the right hand (Edelman et al. 2014, 2015a, b). Wang et al. proposed a MI paradigm that requires imagining different force loads of the right-hand clench (Wang et al. 2017).

Another motor-related paradigm is called movement-related cortical potential (MRCP) that occurs in the planning and execution of movements with a slow decrease in the EEG amplitude lasting at least 500 ms (Shibasaki and Hallett 2006). MRCP comprises three components called readiness potential, motor potential, and movement-monitoring potential, which are considered to reflect movement preparation, execution and performance, respectively (Shibasaki et al. 1980; Hallett 1994; Toro et al. 1994).

MRCP can be extracted from low-frequency (<2 Hz) EEG signals, which contains kinematic information about imagined continuous movement (Waldert et al. 2008; Bradberry et al. 2009). For example, Gu et al. (2009) explored the possibility of simultaneously identifying imagined wrist movements and speeds by MRCP, showing that the rebound rate of MRCPs may indicate the speed of movements. Bradberry et al. (2010) instructed subjects to execute center-out reaching tasks, proving that it can decode 3D hand velocity from EEG signals. They further designed a paradigm requiring subjects to imagine moving right arm/finger to track a computer-controlled 2D cursor (Bradberry et al. 2011). Kim et al. (2014) decoded the 3D trajectory of imagined right arm movements and discussed eye-movement contamination problem in MRCP paradigms. Schwarz et al. (2017) proposed a paradigm to discriminate three reach-and-grasp actions from EEG signals, achieving more natural control of BCI systems. Recently, Wang et al. (2020) investigated the possibility of decoding movement intention before actual movements, showing promising results for developing a new MRCP-based paradigm.

Instead of motor-related functions, the steady-state somatosensory evoked potential is a paradigm focusing on sensor-related functions. Muller-Putz et al. (2006) designed an SSSEP-based BCI paradigm that vibratory stimulations are performed on index fingers. Subjects are instructed to focus their attention to the target finger and count appearing twitches, causing greater spectral amplitude in the target frequency (Giabbiconi et al. 2007). Breitwieser et al. (2012) further applied SSSEP to five fingers of the right hand and proved that it can classify different fingers with SSSEP. Su et al. (2020) explored the relation between stimulation intensities and SSSEP. Their results revealed that SSSEP amplitude is positively correlated with the intensity of stimulation and could be used as an index to evaluate the tactile acuity.

Vision-related paradigms

Vision-related paradigms are important in EEG-based BCIs. These paradigms can modulate neural signals generated from the primary visual cortex. Steady-state visually evoked potential (SSVEP) is such a vision-related paradigm.

The steady-state visually evoked potential is a type of visual evoked potentials (VEPs) found by Regan (1966). SSVEP can be considered as an exogenous event-related potential (ERP) that depends on physical features of sensory stimulus. In SSVEP, a sinusoidally flickering visual stimulus elicits a stable VEP of small amplitude with the corresponding flickering frequency (Vialatte et al. 2010).

The early SSVEP-BCIs were developed based on light-emitting diode (LED) sources (McMillan et al. 1995; Middendorf et al. 2000; Gao et al. 2003). Cheng et al. (2002) implemented SSVEP on a cathode ray tube (CRT) monitor with 13 flickering virtual buttons. Wu et al. (2008) compared SSVEP evoked by LED, CRT and liquid crystal display (LCD), suggesting choosing LED for a highly complicated BCI. But nowadays, most SSVEP experiments are implemented on LCD monitors, due to its high refresh rate, simplicity for designing complex stimulation patterns and reliability to reproduce stable stimulus (Cecotti et al. 2010). In terms of paradigm design, Jia et al. (2010) proposed a joint frequency and phase modulation scheme to increase the number of available stimuli. Nakanishi et al. (2014) implemented a 32-target SSVEP paradigm based on this coding strategy, which was further used to implement a 40-target SSVEP paradigm (Nakanishi et al. 2017a). Instead of using a rectangle stimulus array, Min et al. (2016) designed an SSVEP paradigm with a grid-shaped line array. Instead of gazing at a single flickering stimulus at a time, Tang et al. (2019) designed the multi-focal SSVEPs (mfSSVEPs) paradigm that requires the subject to receive multiple flickers with different frequencies simultaneously. They further designed an mfSSVEPs paradigm containing 32 targets, each comprised 5 flickers flashing at different frequencies (Tang et al. 2020b).

Traditional SSVEP paradigms prefer large-size stimuli in the fovea vision, which can easily cause visual fatigue. Xu et al. (2018) recently designed a new BCI speller based on miniature asymmetric VEPs (miniature aVEPs). The visual stimuli in the miniature aVEP BCI are placed outside the fovea vision on the lateral side, which only occupies $0.5^{\circ }$ of visual angle and induce miniature potentials about $0.5\mathrm{\mu }\mathrm{V}$ in amplitude, showing a promising way to achieve a more comfortable and natural BCI system. They further proposed a Steady-State asymmetrically Visual Evoked Potential (SSaVEP) paradigm with 4 high-frequency stimuli placed outside the fovea vision, achieving an average information transfer rate about 87.2 bits/min for 10 encoded commands (Yue et al. 2020).

Cognition-related paradigms

Comparing to vision-related paradigms, many cognition-related paradigms have been developed for the past few decades. Here we mainly introduce the P300 paradigm, the error-related potentials (ErrPs) paradigm and covert visuospatial attention (CVSA) paradigm.

P300 is a kind of event-related potentials (ERPs) first reported in 1967 (Sutton et al. 1967) and introduced into the BCI community by Farwell and Donchin (1988). P300 is considered as an endogenous ERP that is related to cognition processes without relying on physical features of sensory stimulus. The P300 component can be induced in an oddball paradigm containing two types of stimuli, namely target and non-target stimulis, where the target stimulus appears less frequently than the non-target one. Subjects are required to pay attention to the target stimulus and the infrequent target stimulus elicits a larger positive peak, named the P300 peak, about 300 ms after stimulus onset, compared to the frequent non-target stimulus (Fazel-Rezai et al. 2012). The generation of P300 is independent of sensory pathways regardless of visual and auditory stimuli, although visual stimuli are more commonly used for P300-based BCIs.

The vision-based P300 paradigm is usually used for BCI speller systems. The first P300 speller is a $6\times 6$ matrix of characters and each row and column are flashed in a random order, which is called the row column (RC) paradigm (Farwell and Donchin 1988). The subject is supposed to pay attention to the target character and the flashes of row and column containing that character can elicit the P300 component whereas the others cannot, which makes it possible to detect the target character based on the indexes of row and column. Guan et al. proposed a single character (SC) paradigm which randomly flashes one character at a time (Guan et al. 2004). The SC paradigm is slower than the RC paradigm but results in larger P300 amplitudes (Guger et al. 2009). In order to overcome double target item flash and distraction problems in the RC paradigm (Fazel-Rezai et al. 2012), Townsend et al. (2010) designed a checkerboard paradigm that separates adjacent items into two groups. Fazel-Rezai and Abhari (2009) introduced a hierarchy design into the P300 paradigm and proposed a region-based (RB) paradigm using two levels to recognize the target character. Acqualagna et al. developed a rapid serial visual presentation (RSVP) paradigm presenting all characters sequentially at a single central location independent of gaze shifts, which can also elicit the P300 component (Acqualagna et al. 2010; Acqualagna and Blankertz 2013). Lin et al. (2018) further extended the single-character presentation to triple-character presentation in RSVP.

Recently, Xu et al. (2020d) designed an innovative time estimation paradigm, in which participants predicted the visual stimulus occurs at 400 ms or 600 ms after the cue onset. They found both the time-domain and frequency-domain features were related to time estimation (Meng et al. 2020). Specifically, in the time domain, there were positive P300-like deflections after the predicted moment. The observation is consistent with an explanation that the positive deflections constitute a P300 response to the processing of non-occurrence information, and is similar to the omitted stimulus paradigm that grew out of (Sutton et al. 1967), which reported that the non-occurrence of an auditory stimulus would produce a P300. In this study, the energy features in high-frequency were found to be related to time estimation process, which may provide new neural evidence supporting the hypothesis that the P300 is a multifaceted electroencephalographic response with characteristic features in the frequency domain as well as in the time domain (Farwell and Smith 2001; Farwell 2012). The results reported by Xu et al. (2020d) and Meng et al. (2020) not only demonstrated the cognitive EEG feature elicited by time estimation is possible to work as a novel signal for active BCIs, but also may have discovered a new pattern that could potentially enhance the performance of P300-based BCIs by introducing detection and analysis of frequency-domain features.

The error-related potentials are a series of ERP components evoked after subjects realize they committed errors or receive feedback to errors, e.g. error-related negativity (ERN) (Falkenstein et al. 1991, 2000), feedback-related negativity (FRN) (Frank et al. 2005; Cohen et al. 2007). ERN was found in choice-reaction tasks, which is a negative potential peaking at 50–100 ms after an erroneous response (Falkenstein 1990; Falkenstein et al. 1991; Gehring et al. 1993). ERN can also be evoked when a subject observes another person making an error (van Schie et al. 2004). FRN occurs at 200–300 ms after receiving feedback during a learning task (Holroyd and Coles 2002).

It has been found that ErrPs are quite complex and their latencies differ depending on the tasks (Iturrate et al. 2013). ErrP-based BCIs usually utilize all ErrPs instead of analyzing a certain type of them (Chavarriaga et al. 2014). Ferrez et al. confirmed that ErrPs exist after feedback of incorrect responses in a human-robot interaction experiment and succeeded to detect ErrPs in a single trial (Ferrez and Millán 2005, 2008). Chavarriaga and Millán (2010) further incorporated ErrPs in an automatic cursor’s movement system to correct erroneous movements by detecting ErrPs. Salazar-Gomez et al. (2017) built a closed-loop robotic control system with ErrPs detection to correct mistakes.

Covert visuospatial attention (CVSA), which refers to the process of focusing attention on different regions of the visual field without overt eye movements (Posner 1980), has also been applied in BCI applications. On the one hand, CVSA can be used to improve the performance of other paradigms. It has shown that CVSA can enhance N2 and P3 components in P300-based BCI spellers (Treder and Blankertz 2010). Zhang et al. (2010) designed a SSVEP-based BCI system, in which subjects were instructed to focus attention on two kinds of flashing dots without overt eye movements, showing that the amplitude at the corresponding frequency can also be enhanced by paying attention to one of the two stimuli. Xu et al. (2016) designed a fast CVSA detection paradigm with the N2pc and SSVEP features, achieving an average accuracy of 72.9% by using a data length of 400 ms. Wai et al. (2020) further studied the differences in SSVEP with CVSA and concluded that reliable SSVEP responses can be obtained with covert attention regardless of visual angles and stimulus spatial resolution. On the other hand, Tonin et al. (2013) proposed a BCI based on pure CVSA, requiring subjects to keep covert attention on two different orientations and demonstrating the feasibility of designing BCIs based on CVSA. Gaume et al. (2019) measured continuous vsiual sustained attention in a motor control task, proving that it is possible to estimate task difficulty with the level of attention. Recently, Ahmadi et al. (2020) proposed a covert attention paradigm based on changes in lumination to two colors on two orientations, achieving 91.87% classification accuracy in two-class scenario.

Hybrid paradigms

Hybrid paradigms combine two or more different physiologic signals with at least one EEG channel (Pfurtscheller et al. 2010a; Banville and Falk 2016). Hybrid paradigms aim at improving the performance of the system by utilizing multiple subsystems. A hybrid paradigm could be a combination of BCI paradigms or a BCI paradigm based on EEG and other physiological signals, such as EMG. Below is a review of BCIs that combine multiple paradigms.

Hybrid BCIs usually combine paradigms designed for eliciting different brain functions. Allison et al. (2010) designed a hybrid paradigm combining MI and SSVEP, which requires subjects gazing at SSVEP stimuli and imagining movements simultaneously. Instead of a simultaneous hybrid paradigm, Pfurtscheller et al. proposed an MI-SSVEP hybrid paradigm with switching between paradigms sequentially (Pfurtscheller et al. 2010b). Long et al. combined MI and P300 to control a 2D cursor (Li et al. 2010; Long et al. 2011) and a wheelchair (Long et al. 2012) by asking subjects to imagine movements or pay attention to the flickering stimulus. There are also hybrid paradigms combining MI and SSSEP. Yao et al. asked subjects to imagine left/right hand movements or pay attention to vibrations on the wrists sequentially. They found that it could improve the left and right classification performance by mixing both paradigms (Yao et al. 2013). Yi et al. (2017) performed MI with electrical stimulation on both wrists simultaneously, improving the performance significantly. Recently, Mousavi et al. (2020) proposed a hybrid paradigm combining MI and ErrPs, showing significantly improved performance in online BCI systems.

There are also hybrid paradigms combining paradigms for the same brain function. Panicker et al. (2011) combined P300 and SSVEP to develop an asynchronous P300-speller in which SSVEP was used to recognize subjects’ control states. Xu et al. (2013a) incorporated the SSVEP blocking (SSVEP-B) features into the P300, which significantly enhances the performance of the speller. However, in these studies, users can only focus on one paradigm at a time. Li et al. (2013) proposed a hybrid paradigm by switching part of the stimulus to P300 paradigm transiently in the SSVEP paradigm. Yin et al. (2013a) proposed a 32-character speller with a flickering stimulus presented in the P300 paradigm and further designed two hybrid modes of the stimulus configuration (Yin et al. 2013b). Xu et al. (2013b) designed a new hybrid paradigm evoking SSVEP-B and P300 concurrently and further implemented a large-size high-speed speller with over 100 commands (Xu et al. 2020c).

Decoding algorithms

Decomposition algorithms

Decomposition algorithms are a group of algorithms that use matrix factorization or extract spatial filters to increase the separability of different classes for BCIs. Most decomposition algorithms are designed for feature extraction, which is usually connected to a classifier such as support vector machine (SVM). Decomposition algorithms have been well-developed and still widely used in the BCI community. Most decomposition algorithms can be transformed into optimization problems with constraints and finally solved with generalized eigenvalue decomposition (GED). The form of cost function varies depending on the attributes of encoding paradigms, e.g., whether the encoded information is mainly in the time or time-frequency domain of the EEG signals.

Independent Component Analysis (ICA) is a class of algorithms for blind source separation algorithms that have been widely used to analyze EEG signals (Makeig et al. 1996). ICA can decompose scalp recordings, mixtures of source activities, into the independent components, making it suitable for removing artifact signals, e.g. eye blinks (Jung et al. 2000; Winkler et al. 2014; Frølich et al. 2015; Radüntz et al. 2017). Recently, Chang et al. evaluated the efficacy of combining artifact subspace reconstruction (ASR) and ICA for removing artifactual signals both for offline and online EEG applications (Chang et al. 2019).

Common spatial patterns (CSP) (Ramoser et al. 2000) is the well-known feature extraction method for MI, aiming at maximizing the variances of two distributions of EEG signals, e.g. left and right MI. More details about CSP and its relation to GED can be found in Parra et al. (2005) and Haufe et al. (2014). One of two drawbacks of CSP is that its performance is depending on the selection of frequency band while the optimal frequency band of each subject is not consistent. The other is that CSP is not able to deal with the multi-class scenario. To address the first problem, Ang et al. (2008) proposed a filter bank CSP (FBCSP) that applies CSP to multiple frequency bands and selects features based on the mutual information criterion. For a multi-class scenario, Grosse-Wentrup and Buss (2008) proposed a multi-class CSP framework with information theoretic feature extraction and explored its relation to ICA. Chin et al. (2009) extended the FBCSP to the multi-class scenario by decomposing the multi-class problem into several binary-class problems. The new trend of CSP-based methods is to optimize spectral, temporal and spatial features simultaneously via adding more constraints to the cost function. Higshi et al. proposed discriminative filter bank CSP (DFBCSP) with optimizing filter coefficients and spatial weights, showing that DFBCSP can extract the bands related to MI (Higashi and Tanaka 2012). Meng et al. (2014) further optimized filter coefficients and spatial filters under the framework of mutual information. Zhang et al. (2018d) proposed temporally constrained sparse group spatial pattern (TSGSP) for simultaneously optimizing filter bands and time window. Gurve et al. (2020) proposed Non-Negative Matrix Factorization (NMF) to select subject-specific channels, giving better performance and less number of channels for lower limb MI tasks. Recently, Jin et al. (2020) introduced Dempster-Shafer theory into CSP methods, showing a way to fuse multiple methods by considering the distribution of features.

For SSVEP-based BCIs, Lin et al. (2006) introduced canonical correlation analysis (CCA) to classify SSVEPs. Chen et al. further extended CCA to filter bank CCA (FBCCA), achieving a high-speed BCI speller (Chen et al. 2015a, b). Nakanishi et al. (2017a) proposed task-related component analysis (TRCA) and ensemble TRCA (eTRCA) for SSVEP by maximizing the reproducibility of brain activities across trials. eTRCA further improves the performance of the SSVEP-based spellers comparing to FBCCA. Zhang et al. proposed correlated component analysis (CORCA) by extracting maximally correlated components from different subjects, achieving similar results comparing to eTRCA (Zhang et al. 2018b, c). Jiang et al. (2018) incorporated a dynamic stopping strategy into SSVEP-based BCIs with variable stimulation time of each trial instead of fixed time. Tang et al. (2020a) further designed an modified TRCA (mTRCA) algorithm with the dynamic stopping strategy for a practical online spelling system. For limited calibration data, Wong et al. extended eTRCA to multi-stimulus eTRCA (ms-eTRCA) with a new learning scheme (Wong et al. 2020a) and proposed a unified framework to explain most spatial filtering algorithms for SSVEP-based BCIs (Wong et al. 2020b).

For P300-based BCIs, Serby et al. (2005) used ICA to separate the P300 source(s) from the background noise. Krusienski et al. (2008) proposed stepwise linear discriminant analysis (SWLDA) to construct a classifier, exploring the effects of spatial information on classification. Rivet et al. (2009) designed an unsupervised algorithm named xDAWN to enhance the SNR of evoked potentials. Recently, Xiao et al. (2019) designed a discriminative canonical pattern matching (DCPM) algorithm by combining spatial pattern extraction and pattern matching together, which outperformed other methods significantly in 5 public datasets.

Riemannian geometry

Manifold algorithms, especially Riemannian geometry algorithms, have aroused great interest in the BCI community in the last ten years. The Riemannian geometry applies operations on the space of symmetric positive-definite (SPD) matrices and provides a unified framework to deal with different BCI paradigms by considering each trial as a point in the SPD space (see Congedo et al. (2017) for details).

Barachant et al. concluded basic Riemannian operations, e.g. the Riemannian distance, and proposed the minimum distance to mean (MDM) and MDM with geodesic filtering (FgMDM) algorithms to classify MI tasks (Barachant et al. 2010b). MDM is similar to the nearest neighbor algorithm which uses the Euclidean distance instead of the Riemannian distance. FgMDM projects covariances into the tangent space, applies linear discriminant analysis (LDA) to tangent vectors and then projects them back into the SPD space with selected components. They also investigated the relation between CSP and Riemannian geometry and proposed a method to select CSP components instead of using a heuristic selection method (Barachant et al. 2010a). Due to the scalability of the Riemannian framework, it can be easily expanded to the multi-class scenario and combined with kernel-based machine learning methods (Barachant et al. 2011, 2013). For decoding 6 MI tasks from the same upper limb, Chu et al. (2020) extracted tangent space features followed by partial least squares regression to select more robust features. Considering the complexity of optimizing high dimensional covariance-based cost function, Xu et al. (2020a) studied spatial filters in the tangent space and proposed a dimension reduction method which could achieve the similar performance with reduced computational time cost compared to traditional ones.

Riemannian geometry can also be used to time-series signals evoked by paradigms like SSVEP and P300. Barachant and Congedo (2014) used MDM for P300 by building embedded covariance matrices. Kalunga et al. (2016) applied Riemannian geometry to the SSVEP paradigm and proposed an online implementation method.

Deep learning

Recently, deep learning has been successfully applied in many fields, such as computer vision and nature language processing. Deep learning has also been introduced into the BCI community for its better feature representation ability.

CNN architectures are very popular in the BCI community. Schirrmeister et al. (2017) designed ShallowConvNet and DeepConvNet which imitate temporal and spatial filters in FBCSP, achieving at least as good performance as FBCSP. Vernon et al. further proposed EEGNet that replaces vanilla convolution with depth-wise separable convolution (Lawhern et al. 2018) and validated its performance for SSVEP (Waytowich et al. 2018). Liu et al. (2018) showed that a CNN model with batch normalization could also be used for identifying P300 signals. Dai et al. (2020) proposed a hybrid-scale CNN architecture using different convolution scales in the temporal convolution layer. Some researchers have also explored RNN or CNN-RNN architectures for BCIs. Maddula et al. (2017) used RNN for classifying P300 signals. Zhang et al. (2018a) further designed a cascade convolutional RNN architecture for the MI paradigm. Wang et al. (2018) built a network based on LSTM with inputs of time-frequency features. Attia et al. (2018) used CNN-LSTM architecture to classify SSVEP stimuli. Recently, instead of end-to-end network designs, Ravi et al. (2020) proposed a CNN network with complex spectrum features as inputs and compared its performance with FBCCA and TRCA, showing an improved performance in both user-dependent and use-independent training scenarios. Ma et al. (2020) preprocessed raw EEG data with optimal band selection and PSD feature extraction, improving classification accuracy with reduced number of parameters. Xing et al. (2020) designed a CNN-based comparing network inspired by the standard CCA to learn the relationship between EEG data and templates. Their results suggest that combining the comparing network and TRCA can further improve the performance of SSVEP-based BCIs.

Some other researches focus on data augmentation for BCIs. Abdelfattah et al. (2018) designed a recurrent generative adversarial network (RGAN) to augment movement-related data. Lee et al. further designed C-LSTM model for augmenting MI data (Freer and Yang 2020).

Transfer learning

Many machine-learning algorithms assume that the training and test data are drawn from the same feature space and the same distribution. In BCI, although many algorithms can achieve good performances on a single subject, the practicability of BCIs is still limited to the high variability of EEG signals from different subjects or sessions. These problems are referred to as cross-subject and cross-session variability problems. To alleviate the influence of these two problems, a calibration stage is usually required to collect enough training data at the beginning of each session, which is inconvenient for both patients and healthy subjects. Transfer learning aims to improve the learning process of the predictive function in the target domain using the knowledge in the source domain (Pan and Yang 2009), which makes it suitable for solving cross-session and cross-subject variability problems.

The early transfer-learning algorithms for BCIs focus on the improvement of decomposition algorithms. Krauledat et al. (2008) extracted stable prototype filters from different sessions of the same subject to reduce calibration time for a new session. Kang et al. (2009) proposed a composite CSP with regularized covariance matrices for subject-to-subject transfer. Lotte et al. further improved the regularization strategy and proposed a unified regularization framework for CSP (Lotte and Guan 2010a, b). Wang et al. (2012) used ICA to transfer spatial filters from the resting state to MI tasks. Another way to reduce variability is to use stationary subspace analysis (SSA). Bunau et al. proposed SSA to decompose a multivariate time series into its stationary and nonstationary components, using only stationary components to classify tasks (Von Bünau et al. 2009). The SSA method finds a projection matrix that projects the data onto a stationary subspace by optimizing a cost function based on the estimation of matrix divergence, namely Kullback–Leibler divergence (KL-divergence). Horev et al. (2016) used the Riemannian distance to measure the non-stationary characteristic. Kaltenstadler et al. (2018) further improved this method by using Wasserstein distance (WaSSA). Recently, Chiang et al. (2020) proposed a transfer learning framework with least-squares transformation (LST) for SSVEP BCIs, showing promising results for cross-subject and cross-device scenarios.

Riemannian geometry has promoted the development of transfer-learning algorithms for BCIs. In 2011, Reuderink et al. (2011) presented a second-order baselining procedure to reduce variabilities in BCIs by using a pre-trial baseline covariance matrix to whiten task covariance matrices. The whitening idea has been further developed based on Riemannian geometry. Instead of a pre-trial baseline covariance matrix, Zanini et al. (2017) used the Riemannian mean covariance matrix as the reference matrix and transformed the center of all covariance matrices to the identity matrix. This procedure was applied for each session or subject, respectively. To reduce its computational cost, He and Wu (2019) suggested using the Euclidean mean covariance matrix instead of the Riemannian mean covariance matrix. Li et al. (2020) validated the xDAWN algorithm combined with Riemannian whitening for P300 datasets. Qi et al. (2018) proposed a speedy calibration method with Riemannian geometry for P300 spellers by selecting related samples from the database. Rodrigues et al. (2018) proposed a Riemannian Procrustes Analysis (RPA) with translation, scaling, and rotation transformations. To apply RPA for heterogeneous datasets, they further proposed a method for merging datasets with different numbers of electrodes by adding the white noise to the empty channel (Rodrigues et al. 2020).

Deep learning can also be used for solving variability problems in BCIs. Sakhavi and Guan (2017) transferred pre-trained CNN to a new subject with fine-tuning and knowledge distillation. ShallowConvNet and EEGNet also showed the cross-subject generalization ability in single datasets without fine-tuning (Schirrmeister et al. 2017; Lawhern et al. 2018). Xu et al. (2020b) further found that deep learning models with Riemannian geometry could significantly improve the generalization ability across BCI datasets without any additional calibration data.

Researchers have also studied some transfer learning algorithms from other fields. Waytowich et al. (2016) proposed spectral transfer using information geometry (STIG), which was validated in the RSVP paradigm. Yair et al. considered the cross-subject problem for MI by using parallel transport (PT) on the SPD manifold, giving an extension of the Riemannian whitening method (Yair et al. 2019a; Maman et al. 2019). They further improved the results by using optimal transport (OT) (Yair et al. 2019b). Zhang and Wu (2020) proposed a manifold embedded knowledge transfer (MEKT) method for BCIs by fusing the transfer component analysis (TCA) (Pan et al. 2010) and the joint distribution adaptation (JDA) (Long et al. 2013).

Conclusion

In this review, we have surveyed recent progress in BCIs from the perspectives of encoding paradigms and decoding algorithms. Encoding paradigms are divided into four categories based on their related brain functions: sensory- and motor-related paradigms, vision-related paradigms, cognition-related paradigms and hybrid paradigms. MI and other paradigms focusing on the somatosensory and motor areas are important in the BCI field. Meanwhile, vision-related paradigms develop rapidly and many new paradigms have emerged in recent years. Cognition-related paradigms could improve the robustness of BCI systems with other paradigms. Hybrid paradigms have also shown promising potential in developing efficient BCI systems. Decoding algorithms are grouped into four families: decomposition algorithms, Riemannian geometry, deep learning and transfer learning. Decomposition algorithms are quite effective in dealing with the limited number of samples in supervised scenario. Riemannian geometry provides a unified framework to process EEG signals from different paradigms. Deep learning models have shown advantages in large-size samples. Recently, transfer learning has attracted much attention from the BCI community due to its ability to solve human variability problems in BCIs. By combining these paradigms and algorithms, a more robust and effective BCI system may be developed in the future.

Author Contributions

LX and MX contributed equally to the study conception, literature search, and writing. All authors contributed to manuscript revision, read and approved the submitted version.

Funding

This research was funded by National Key Research and Development Program of China, Grant No. 2017YFB1300300; National Natural Science Foundation of China, Grant Nos. 81925020, 61976152, 81671861; Young Elite Scientist Sponsorship Program by CAST, Grant No. 2018QNRC001.

Declarations

Conflict of interest

The authors declare that they have no conflict of interest.