Filter bank temporally local canonical correlation analysis for short time window SSVEPs classification

Abstract

Canonical correlation analysis (CCA) method and its extended methods have been widely and successfully applied to the frequency recognition in SSVEP-based BCI systems. As a state-of-the-art extended method, filter bank canonical correlation analysis has higher accuracy and information transmission rate (ITR) than CCA. However, in the CCA method, the temporally local structure of samples has not been well considered. In this correspondence, we proposed termed temporally local canonical correlation analysis (TCCA). In this new method, the original covariance matrix was replaced by the temporally local covariance matrix. Furthermore, we proposed an improved frequency identification method of filter bank based on TCCA, named filter bank temporally local canonical correlation analysis (FBTCCA). In the offline environment, we used a leave-one-subject-out validation strategy on datasets of ten testees to optimize the parameters of TCCA and FBTCCA and evaluate the two algorithms. The experimental results affirm that TCCA markedly outperformed CCA, and FBTCCA obtained the highest accuracy among the four methods. This study corroborates that TCCA-based approaches have great potential for implementing short time window SSVEP-based BCI systems.

Introduction

Brain–computer interface (BCI) is an emerging method that could provide a new communication channel between a human and an external environment (Lance et al. 2012; Chaudhary et al. 2016). In a BCI system, noninvasive EEG has been valued by researchers because of its safety and low-cost devices (Lay-Ekuakille et al. 2013; Dai et al. 2015; Chen et al. 2016; Wang et al. 2016). There are several brain control signals commonly used in BCI systems, such as event-related potential (ERP) (Cheng et al. 2002; Gao et al. 2014; Miao et al. 2020), sensorimotor rhythm (SMR) (He et al. 2015; Zhang et al. 2017; Feng et al. 2018), auditory steady-state response (ASSR) (Hwang et al. 2017),steady-state visual evoked potential (SSVEP) (Yin et al. 2014; Chen et al. 2015b; Zhang et al. 2015; Chang et al. 2016; Jiao et al. 2018), etc. The SSVEP-based BCI is one of the core areas of research in the field of BCI, and studies indicated that it has the preponderance of high information transmission rate and less user training (Cheng et al. 2002; Wang et al. 2008; Bakardjian et al. 2010; Gao et al. 2014).

SSVEPs are generated by the users staring at the visual flickers and include the same frequency, harmonic, and subharmonic brain responses as the stimulus (Herrmann 2001). The development directions of SSVEPs detection method are shorter the time window, higher the accuracy, and ITR. Various advanced signal processing methods have been proposed to detect SSVEPs, the power spectrum density analysis (PSDA) (Hwang et al. 2012), canonical correlation analysis (CCA) (Lin et al. 2006), likelihood ratio test (LRT) (Zhang et al. 2014a), multivariate synchronization index (MSI) (Zhang et al. 2014b) and correlated component analysis (CORRCA) (Zhang et al. 2018) are the mainstream algorithms. Among various methods, the CCA method proposed by Lin plays a compelling role in the detection of SSVEPs due to its high efficiency and robustness (Lin et al. 2006). In order to improve the performance of CCA, many extended methods are proposed (Zhang et al. 2013b; Poryzala and Materka 2014; Zhang et al. 2014c; Chen et al. 2015a; Nakanishi et al. 2015), such as the L1-regularized multiway CCA (L1-MCCA) (Zhang et al. 2013b), cluster analysis of CCA coefficient (CACC) (Poryzala and Materka 2014), multiset CCA (MsetCCA) (Zhang et al. 2014c), filter bank CCA (FBCCA) (Chen et al. 2015a), a combination method of CCA and IT-CCA (Nakanishi et al. 2015). The FBCCA method takes into account the information contained in harmonics, which significantly improves the performance of BCI based on SSVEP. Under 40 target frequencies, the average ITR of the online spelling system based on FBCCA was 151.18 $\pm 20.34$ bits ${\text{min}}^{-1}$ (Chen et al. 2015a).

Owing to the EEG signal is non-stationary, it will change slowly with time (Muller et al. 2003), and temporally local variances may provide more information (Wang and Xu 2012). Previous researches have proved the positive effect of temporally local information, such as TMSI (Zhang et al. 2016), TLMSFA (Wang 2010), LTCSP (Zhang et al. 2013a). Mathematically, the calculation of CCA depends on the covariance matrix, and the CCA-based methods ignored the temporal information of signals.

In this study, temporally local information was considered in the CCA algorithm for the first time, and a new method called TCCA was proposed. On this basis, we combined the two techniques of TCCA and FBCCA and recommended the FBTCCA method. The datasets of ten subjects were used for a leave-one-subject-out validation strategy. The parameters of TCCA and FBTCCA were optimized on the training set. The target recognition accuracy of CCA, TCCA, FBCCA, and FBTCCA was compared on the test set. Finally, the results proved that the optimized TCCA is significantly better than the CCA, and FBTCCA has the best performance among the four algorithms. The TCCA-based algorithm has excellent potential in SSVEP frequency identification, especially in a BCI system with a short time window.

Materials and methods

Standard CCA

CCA is a tool to measure the similarity between two multivariate variables, which is widely used in SSVEPs detection. Given two sets of variables $X\in {\mathbb{R}}^{M\times K}$and $Y\in {\mathbb{R}}^{2N\times K}$. CCA seeks two corresponding weight vectors ${\omega }^{M\times 1}$ and ${\upsilon }^{N\times 1}$ to maximize the correlation coefficient of X and Y:

$$\begin{array}{c}\hfill \rho =arg\underset{\omega ,\upsilon }{\mathit{max}}\frac{{\omega }^{T}X{Y}^T\upsilon }{\sqrt{{\omega }^{T}X{X}^T\omega }\sqrt{{\upsilon }^{T}Y{Y}^T\upsilon }}\end{array}$$ 1

$\rho$ is the maximum correlation coefficient between linear combination ${\omega }^{T}X$ and ${\upsilon }^{T}Y$.$X\in {\mathbb{R}}^{M\times K}$ and $Y\in {\mathbb{R}}^{2N\times K}$ respectively represent multi-channel SSVEPs and reference signals. M is the number of signal channels, N denotes the number of harmonics and K denotes the number of time samples. The reference signal $Y\in {\mathbb{R}}^{2N\times K}$ is manually constructed as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Y=Y_n=\left[\begin{array}{c}sin(2\pi f_{n}t)\\ cos(2\pi f_{n}t)\\ ⋮\\ sin(2\pi N{f}_{n}t)\\ cos(2\pi N{f}_{n}t)\end{array}\right],t=\frac{1}{F_S},\frac{2}{F_S},\dots ,\frac{K}{F_S}\end{array}\end{array}$$ 2

where $f_n$ is the nth stimulus frequency, each stimulus frequency corresponds to the commensurable reference signal. When the number of stimulation frequencies is S, the corresponding stimulation frequency can be estimated from the maximum value of the correlation coefficient between each reference signal and the test sample:

$$\begin{array}{c}\hfill f_{\mathit{test}}=\underset{f_n}{\mathit{max}}{\rho }_n,n=1,2,\dots ,S\end{array}$$ 3

Temporally local CCA

It has been proved that considering the time structure information of EEG can remove the artifacts and improve the accuracy of the signal processing algorithm. We define an adjacency matrix $A\in {\mathbb{R}}^{K\times K}$ based on the number of samples:

$$\begin{array}{c}\hfill A(i,j)=W(\frac{j-i}{\tau })\end{array}$$ 4

And the value of W is determined by Tukey’s tricube weighting function:

$$W(\frac{j-i}{\tau })=\left\{\begin{array}{cc}(1-{\left|\frac{i-j}{\tau }\right|}^3{)}^3,\hfill & \left|\frac{i-j}{\tau }\right|<1\hfill \\ 0,\hfill & \text{else}\hfill \end{array}\right)$$ 5

Multiple signals can also be written $X=[x_1,x_2,\dots ,x_K]\in {\mathbb{R}}^{M\times K}$. The formula of temporally local covariance matrix is as follows:

$$\begin{array}{c}\hfill \overline{C}=\frac{1}{2K}\sum _{i=1}^K\sum _{j=1}^{K}A(i,j)(x_i-x_j){(x_i-x_j)}^T\end{array}$$ 6

Formulas (6) can be converted to:

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill \overline{C}& =\frac{1}{K}\sum _{i=1}^K{x}_i{x}_i^T\sum _{j=1}^{K}A(i,j)-\sum _{i=1}^K\sum _{j=1}^{K}A(i,j)x_i{x}_j^T\hfill & \\ \hfill & =\frac{1}{K}(X(D-A)X^T)\hfill & \\ \hfill & =\frac{1}{K}(XL{X}^T)\hfill & \\ \hfill \end{array}\end{array}$$ 7

D is a diagonal matrix, and $D(i,i)={\sum }_{j=1}^{K}A(i,j)$ for $i=1,2,\dots ,K$ , $L=D-A$ is Laplacian matrix. In this paper, L is used to reconstruct the formula for calculating the maximum correlation coefficient, The new maximum correlation coefficient is:

$$\begin{array}{c}\hfill \rho \prime =arg\underset{\omega \prime ,\upsilon \prime }{\mathit{max}}\frac{\omega {\prime }^{T}XL{Y}^T\upsilon \prime }{\sqrt{\omega {\prime }^{T}XL{X}^T\omega \prime }\sqrt{\upsilon {\prime }^{T}YL{Y}^T\upsilon \prime }}\end{array}$$ 8

If $S_{\mathit{xy}}=XL{Y}^T$,$S_{\mathit{xx}}=XL{X}^T$,$S_{\mathit{yy}}=YL{Y}^T$, the optimization objective is:

$$\begin{array}{c}\hfill \rho \prime =arg\underset{\omega \prime ,\upsilon \prime }{\mathit{max}}\frac{\omega {\prime }^T{S}_{\mathit{xy}}\upsilon \prime }{\sqrt{\omega {\prime }^T{S}_{\mathit{xx}}\omega \prime }\sqrt{\upsilon {\prime }^T{S}_{\mathit{xy}}\upsilon \prime }}\end{array}$$ 9

This question is translated into:

$$\begin{array}{c}\hfill \begin{array}{ccc}& arg\underset{\omega \prime ,\upsilon \prime }{\mathit{max}}\omega {\prime }^T{S}_{\mathit{xy}}\upsilon \prime \hfill & \\ \hfill & s.t.\omega {\prime }^T{S}_{\mathit{xx}}\omega \prime =1,\upsilon {\prime }^T{S}_{\mathit{yy}}\upsilon \prime =1\hfill & \\ \hfill \end{array}\end{array}$$ 10

SVD can solve the optimization objective,let

$$\begin{array}{c}\hfill \omega \prime =S_{\mathit{xx}}^{-1/2}\xi ,\upsilon \prime =S_{\mathit{yy}}^{-1/2}\eta \end{array}$$ 11

By introducing this expression into (10), we get:

$$\begin{array}{cc}& {\xi }^T\xi =1,{\eta }^T\eta =1\hfill \end{array}$$ 12

$$\begin{array}{cc}& \omega {\prime }^T{S}_{\mathit{xy}}\upsilon \prime ={\xi }^T{S}_{\mathit{xx}}^{-1/2}{S}_{\mathit{xy}}{S}_{\mathit{yy}}^{-1/2}\eta \hfill \end{array}$$ 13

Both $\xi$ and $\eta$ are unit orthogonal basis, they are regarded as left and right singular vectors corresponding to a singular value of matrix $M=S_{\mathit{xx}}^{-1/2}{S}_{\mathit{xy}}{S}_{\mathit{yy}}^{-1/2}$.Using singular value decomposition, $M=U\mathrm{\Sigma }{V}^T$.

The formula (14) is maximized to obtain the maximum singular value $\rho \prime$ and the left and right singular vectors corresponding to the maximum singular value. $\omega \prime$ and $\upsilon \prime$ can be obtained by (10).

$$\begin{array}{c}\hfill {\xi }^T{S}_{\mathit{xx}}^{-1/2}{S}_{\mathit{xy}}{S}_{\mathit{yy}}^{-1/2}\eta ={\xi }^{T}U\mathrm{\Sigma }{V}^T\eta \end{array}$$ 14

The above formula (8) is the core idea of temporally local CCA. Temporally local CCA looks for $\omega \prime$, $\upsilon \prime$ to make $\rho \prime$ reach the maximum value. Then we can use formula (3) to find the target stimulation frequency.

FBTCCA

The FBCCA method has been successfully applied to a high-speed BCI Speller. By designing filter banks, FBCCA perfectly utilizes the information contained in harmonics, and its performance is far superior to that of standard CCA. Based on TCCA, we considered the effect of harmonic components and proposed a new method called FBTCCA. The flowchart is shown in Fig. 1.

Fig. 1.

Flowchart of the FBTCCA method

First of all, the original EEG signal is decomposed into $N_f$ subbands with different frequencies through the filter bank. The frequency ranges of subbands are 8–88 Hz, 16–88 Hz,$\dots$,$N_f\times$ 8–88 Hz. Then, correlation analysis is carried out for each subband and reference signal, SB1 is analyzed by TCCA, and other subbands are analyzed by CCA. The weighted square sum of the correlation coefficients of each subband is obtained, as shown in the formula (15):

$$\begin{array}{c}\hfill {\tilde{\rho }}_n=\sum _{l=1}^{N_f}q(l)·{({\rho }_n^l)}^2\end{array}$$ 15

where q(l) is the weight of the lth subband component, expressed as:

$$\begin{array}{c}\hfill q(l)=l^{-a}+bl=1,2,\dots ,N_f\end{array}$$ 16

Finally, we use formula (3) to identify the target frequency.

Datasets acquisition

The EEG data in this paper were from the published database (Yuan et al. 2015). We used the SSVEPs dataset of 10 healthy subjects (3 females, aged from 18 to 31 years, all right-handed) participating in the offline experiment. There are 40 different frequency stimuli on the LCD screen (frequency range is 8–15.8 Hz, the interval is 0.2 Hz). For each visual stimulus, subjects watched for five seconds, and this test needs to be repeated six times. Subjects’ eyes were 70 cm from the screen, and they would rest for a few minutes between the two trials to prevent visual fatigue.

The EEG signals of subjects were recorded by a Synamps2 system (Neuroscan, Inc.) system at a sampling rate of 1000 Hz. The preprocessing of EEG includes a 0.15–200 Hz bandpass filter and a notch filter at 50 Hz. The arrangement of electrodes follows the international 10–20 system. We recorded the EEG signals of PZ, PO5, PO3, POZ, PO4, PO6, O1, Oz and O2 channels, all nine channels are located in the occipital region. The researcher placed the reference electrode at CZ and kept the electrode impedance below 10 K$\mathrm{\Omega }$. Finally, all the recorded SSVEPs were down-sampled to 250 Hz.

Parameter optimization and performance evaluation

A leave-one-subject-out validation strategy is used to optimize the algorithm parameters and evaluate the algorithm. From 10 subjects, we extract the data set of each subject in turn as the test set, and the data set of the other nine subjects as the training set. We optimized the parameters of the new algorithms on the training set and evaluated the accuracy on the test set.

In this paper, we fixed the filter number $N_f$ to 5 and set the weight parameters a and B in FBCCA to 1.25, 0.25. Under this condition, FBCCA achieves the highest performance. Therefore, the parameters to be optimized are $\tau$ in TCCA, $\tau$, a, b in FBCCA. We used the grid search method to determine the above parameters. The ratio of the correct number of targets to the total number of targets (i.e., 40) can be regarded as the classification accuracy. After parameter optimization, we divided the four algorithms into two groups for accuracy evaluation. The first group is TCCA and CCA, and the second group is FBTCCA and FBCCA. The time windows are set to 0.5s, 0.6s, 0.7s, 0.8s, 0.9s, 1.0s, 1.2s and 1.5s respectively. The average accuracy of each algorithm under different time windows was compared, and the difference of results was observed by the paired-sample t-test. The efficacy of the proposed algorithm is verified by comparing the r-square value of correlation coefficient for the target frequency versus the remaining frequencies. These four algorithms are all run by MATLAB 2014b on a 2.9 GHz CPU (16G RAM) desktop computer. Statistical analysis between different methods was performed by a paired-sample T-test using the SPSS software package.

Results

The researchers have proved that the choice of the value of the harmonic number N in the reference signal has a considerable influence on the classification accuracy.In this study, N was set to 5 (Chen et al. 2015a). For TCCA, it is necessary to fix a parameter $\tau$, which is determined by a grid search method. $\tau$ started at 2 and ended at 30 with an interval of 2, and the time window was 0.8s. In Fig. 2, When $\tau$ was in the range of [2 30], the accuracy of TCCA was higher than CCA. When $\tau =6$, efficiency reached the highest point, this setting was applied to the comparison between TCCA and CCA. As Fig. 3 shows, the accuracy of TCCA is higher than that of CCA in every time window. In addition, a paired-sample t-test was applied to observe significance. The outcomes indicated that there were significant differences between CCA and TCCA ($P<0.05$). A segment of SSVEP with a visual stimulation frequency of 12.2 Hz was selected. The R-square between SSVEP and 40 frequency reference signals are calculated by CCA and TCCA, respectively. The results are shown in the Fig. 4. When the reference signal is 10.6 Hz, the R-square value of CCA exceeds the R-square value of the target frequency, resulting in a frequency identification error. In the TCCA method, the R-square value of the target frequency reaches the highest.

Fig. 2.

Classification accuracy of TCCA using different temporally local range with the time window being 0.8s

Fig. 3.

Accuracy comparison of TCCA and CCA at different time windows. SE in the form of error bars

Fig. 4.

Visual stimulation frequency = 12.2 Hz. The effect of different stimulation frequency in the reference signal on R-square value in CCA and TCCA

For FBTCCA, three parameters need to be determined, which were time local region ${\tau }^{{}^{\prime }}$, a and b in the weight formula (16). Two grid search methods are used to search for weight parameters and ${\tau }^{{}^{\prime }}$. In this paper, ${\tau }^{{}^{\prime }}$, a, b were respectively limited to [2:2:24], [0.25:0.25:2.5], [0:0.25:1]. Figure 5 shows the classification accuracy of FBTCCA at different ${\tau }^{{}^{\prime }}$, the time window was 0.8s, and the values of a and b were temporarily set as 1.25 and 0.25. In this case, FBCCA has optimized performance (Chen et al. 2015a). When ${\tau }^{{}^{\prime }}$ was in the range [8 24], the accuracy of FBTCCA exceeded that of FBCCA.

Fig. 5.

Classification accuracy of FBTCCA using different temporally local range with the time window being 0.8s

Because the processing method of the first subband in the FBTCCA method is TCCA, we need to determine a and b to make FBTCCA have the best performance, The results in Fig. 6 show that FBTCCA performs best when a = 2.25, B = 0.25. Then the optimized FBTCCA was compared with FBCCA. It was evident from Fig. 7 that when the time window is less than 1s, the classification accuracy of FBTCCA is significantly better than that of FBCCA ($P<0.05$). A segment of SSVEP with a visual stimulation frequency of 9.8 Hz was selected. The sum of R-square between SSVEP and 40 frequency reference signals are calculated by FBCCA and FBTCCA, respectively. The results are shown in the Fig. 8. When the reference signal is 8.2 Hz, the sum of R-square value of FBCCA exceeds the sum of R-square value of the target frequency, resulting in a frequency identification error. In the FBTCCA method, the sum of R-square value of the target frequency reaches the highest.

Fig. 6.

Grid search for optimizing the parameters a, b

Fig. 7.

Accuracy comparison of FBTCCA and TCCA at different time windows. The asterisks symbolize the significant difference between the two methods (one asterisk: $P<0.05$, two asterisks: $P<0.01$, three asterisks: $P<0.001$).SE in the form of error bars

Fig. 8.

Visual stimulation frequency = 9.8 Hz. The effect of different stimulation frequency in the reference signal on the sum of R-square value in FBCCA and FBTCCA

Discussion and conclusion

The short time window can reduce the fatigue of subjects and improve the computing speed for the BCI system based on SSVEP. CCA and its extended algorithm are the mainstream algorithms of frequency recognition in SSVEP, among which FBCCA has excellent performance and still has high accuracy and ITR under 40 targets. However, CCA needs to consider the time information of the signal. TMSI TMSFA and others have proved that finding the time in formation in the signal can improve the performance of the algorithm. To our knowledge, no one has ever thought over the time information in the CCA, to reduce the artifacts and to raise the accuracy of the algorithm under the short time window. For the first time, we took the time information on the CCA method into account. Figure 3 and Table 1 show that the accuracy of TCCA is higher than that of CCA when the temporally local parameters are appropriate. On this basis, we proposed the FBTCCA algorithm by combining the filter bank with TCCA. For FBTCCA, we used the grid search method to determine three parameters. Namely, ${\tau }^{{}^{\prime }}$, a, b, ${\tau }^{{}^{\prime }}$ established the relationship between two data points in the adjacency time range in TCCA, and a, b determined the influence weight of the $\rho$ obtained by each sub-band after passing through the filter bank. These three parameters affect the performance of FBTCCA. In this paper, we consider three parameters, respectively, and hope to find parameters at the same time in future research. Besides, there may be a better way to build a formula (16). Figures 4 and 8 show the efficacy of TCCA and FBTCCA from the perspective of R-square. In terms of time consumption, the calculation speed of the four algorithms on the same computer is as shown in Table 2:

Table 1.

Classification accuracy (%) of four methods at different time windows

TW	Method
	CCA	TCCA	FBCCA	FBTCCA
0.5s	$34.96\pm 21.27$	$41.37\pm 24.81$	$42.08\pm 19.85$	$44.21\pm 2078$
0.6s	$41.13\pm 23.09$	$49.67\pm 26.24$	$52.64\pm 23.31$	$54.26\pm 23.44$
0.7s	$47.67\pm 23.81$	$57.29\pm 24.96$	$61.85\pm 23.13$	$63.89\pm 22.45$
0.8s	$52.71\pm 24.27$	$62.83\pm 23.92$	$70.09\pm 21.91$	$71.57\pm 21.32$
0.9s	$57.71\pm 23.84$	$66.75\pm 23.61$	$74.77\pm 21.47$	$75.83\pm 20.48$
1.0s	$61.04\pm 24.96$	$70.58\pm 23.46$	$78.71\pm 21.28$	$79.49\pm 20.31$
1.2s	$67.75\pm 27.43$	$76.17\pm 23.56$	$85.23\pm 19.35$	$85.69\pm 18.31$
1.5s	$75.17\pm 26.60$	$81.88\pm 20.01$	$90.42\pm 15.73$	$91.16\pm 13.39$

Table 2.

Computational time(s) of four methods at different time windows

TW	Method
	CCA	TCCA	FBCCA	FBTCCA
0.5s	0.0065	0.0681	0.0435	0.105
0.6s	0.0066	0.0827	0.0437	0.1206
0.7s	0.0081	0.1007	0.0518	0.1417
0.8s	0.0078	0.1201	0.0510	0.1563
0.9s	0.0083	0.1407	0.0543	0.1807
1.0s	0.0084	0.1597	0.0519	0.2043
1.2s	0.0107	0.1986	0.0667	0.2700
1.5s	0.0120	0.3507	0.0749	0.4064

If TCCA is applied to all subbands decomposed by filter banks, more computation time is needed. The duration of TCCA processing only for SB1 is completely acceptable.

Table 1 shows the average classification accuracy of 9 testees at different time windows of the four methods. Plainly, FBTCCA has the highest classification accuracy among the four algorithms. Compared with the FBCCA, the accuracy of the FBTCCA is remarkably improved in the short time window. FBTCCA has the highest classification accuracy among the four algorithms. Even when the time window is 1s, and the stimulus target is 40, the FBTCCA algorithm has high accuracy. And because FBTCCA has not low computing speed, it can be inferred that FBTCCA in an online system will also have excellent performance. However, the difference between FBCCA and FBTCCA is very small when the data length over 1s, this is related to the high accuracy of FBCCA itself when the data length is long, and the next work is to improve the performance of FBTCCA in a long time window.

To sum up, we put forward TCCA and FBTCCA methods, combined with the filter bank and time information. The experimental results show that for the accuracy of target classification in a BCI system based on SSVEP, especially in the short time window, The performance of TCCA and FBTCCA is better than that of CCA and FBCCA, respectively. In a BCI system with a short time window, TCCA and FBTCCA have great potential.