INCORPORATING TEMPORAL DEPENDENCY ON ERP BASED BCI

Abstract

In brain computer interface (BCI) systems based on event related potentials (ERPs), a windowed electroencephalography (EEG) signal is taken into consideration for the assumed duration of the ERP potential. In BCI applications inter stimuli interval is shorter than the ERP duration. This causes temporal dependencies over observation potentials thus disallows taking the data into consideration independently. However, conventionally the data is assumed to be independent for decreasing complexity. In this paper we propose a graphical model which covers the temporal dependency into consideration by labeling each time sample. We also propose a formulation to exploit the time series structure of the EEG.

1. INTRODUCTION

Brain computer interfaces (BCIs) are designed to help people with locked in syndrome communicate. Rapid serial visual presentation (RSVP) is a BCI technique in which symbols are shown once at a time on a fixed location at a rapid rate in a random order [1, 2]. RSVP Keyboard™ is a framework which relies on event related potentials (ERPs) for binary classification [3, 4]. It is considered that ERP lasts 500[ms]. Each sequence of rapid flashes has a fixed number of trials (symbol flashes). Between sequences, the time gap is more than 500[ms] which separates sequence data temporally. Bayesian update can be performed between sequences multiplicatively as the sequences are independent. Evidence likelihood in sequences in RSVP scheme is often calculated multiplying the trial scores. Likelihood estimate can be calculated multiplying the trial scores when trial data are statistically independent from each other. However in RSVP scheme inter stimuli interval between flashes is often shorter than 500[ms]. This results in overlaps between trials. Therefore due to temporal dependency trial data is not statistically independent. In this paper we investigate a formulation of the decision making procedure specified for RSVP Keyboard™ which can be generalized for other BCIs by keeping the recursive Bayesian update between sequences and taking temporal dependency into consideration.

To keep the descriptions clear and simple in the paper we summarize the notation used throughout the paper in Tab.1, where the lower notation denotes the order index for the variable’s domain.

Table 1:

Notation Legend

X : Symbol set with cardinality |X|

Si:j : all sequences from ith to jth where j > i

${\epsilon }_{0:k}^{S_i}$: EEG samples in ith seq. between [0, k]

${\mathcal{l}}_{0:k}^{S_i}$: Latent variable in ith seq. between [0, k]

${\mathbf{\Phi }}_{0:k}^{S_i}$: Flashed symbols in ith seq. between [0, k]

${\epsilon }^{S_i}=\left[{\epsilon }_0^{S_i},{\epsilon }_1^{S_i},\cdots ,{\epsilon }_N^{S_i}\right]\in {ℝ}^{N\times 1}$

${\mathbf{\Phi }}^{S_i}=\left[{\mathbf{\Phi }}_0^{S_i},{\mathbf{\Phi }}_1^{S_i},\cdots ,{\mathbf{\Phi }}_M^{S_i}\right]{\mathbf{\Phi }}_m\in X\forall m$

Assume that an intent decision is made after i sequences. Recursive Bayesian update with temporal independence assumption is defined as,

$$\begin{gathered} \widehat{x}=\text{arg}\underset{x}{\text{max}}p(x|{\epsilon }^{S_{0:i}},{\mathbf{\Phi }}^{S_{0:i}}) \\ =\text{arg}\underset{x}{\text{max}}p(x|{\epsilon }^{S_{0:i-1}},{\mathbf{\Phi }}^{S_{0:i-1}})\frac{p\left({\epsilon }_{0:N}^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right)}{p\left({\epsilon }^{S_i}|{\mathbf{\Phi }}^{S_i}\right)} \end{gathered}$$ (1)

where S_i denotes i^th sequence. The time independence simplifies the conditional EEG distribution as,

$$p\left({\epsilon }^{S_i}|x,{\mathbf{\Phi }}^{S_i}\right)\propto \prod _{k}p\left({\mathcal{l}}_k^{S_i}|{\epsilon }^{S_i}\right)$$ (2)

where k denotes the trial index in a particular sequence and ℓ denotes the label of the corresponding binary question. Inserting Eq.(2) into Eq.(1) yields the decision process.

In BCI frameworks binary classification over ℓ in Eq.(2) is performed using support vector machines [5, 6, 7], quadratic and linear discriminators [8, 9] conventionally. These methods assume that for a RSVP paradigm, the data within 500[ms] interval after the symbol flash is related with the intent. In such methods a classification can be performed using the entire data in which the temporal structure of the data is lost and the trial data is assumed to be independent. In RSVP Keyboard™ the classification is performed using regularized discriminant analysis [10] with the assumption of independent trials. With the independence assumption Bayesian update can be performed multiplying scores of trials [11].

However, as we explained before, trial data overlap and the independence, even it eases the calculations, should not be assumed. Therefore it is not convenient to have multiplications denoted in Eq.(2). Temporal dependency can be dealt with by redefining the trial data and separating it to non-overlapping EEG blocks. Using this information different blocks for each EEG yields a multi class classification problem for each trial [12]. This formulation is promising yet complicated. Authors propose to train a set of classifiers for sets of EEG chunks in a one versus all manner. This division does not entirely cover dependency and still has gaps in sequence updates.

In this paper we propose the graphical model in Fig. 1. We label each time sample of EEG data which allows to enforce statistical independence among EEG given the label information. In previous work the independence among EEG data is assumed without statistically representing it with a model and/or is limited with dividing EEG into chunks. We also propose a model to exploit time series structure of the EEG data. In conventional methods likelihood of the labels are computed feeding the entire chunk of EEG into a classifier which neglects the sequentiality in data.

Fig. 1:

(a) Graphical model for a single sequence §_i. Model represents one recursive Bayesian update denoted in Eq.(1). Observe that a single letter flash $\left({\mathbf{\Phi }}_m^{S_i}\right)$ effects multiple label chunks ${\overline{\mathcal{l}}}_{c_i:c_2}^{S_i}$. (b) Definition of label and EEG chunks respectively. Observe that there lies a dependency between labels if the flashed letters are not given. Therefore in (b) labels will have dependencies over each other.

2. METHOD

In RSVP Keyboard™, update in Eq.(1) is performed after a sequence of trials. The sequences are independent of each other. Let us denote the set of sequences within an epoch with S = {S₀, S₁, … S_C}. Within each sequence let us denote the observed EEG and the displayed symbols of i^th sequence as ${\epsilon }^{S_i}$ and ${\mathbf{\Phi }}^{S_i}$ respectively. We denote the prior probability of each letter with p(x), which can be also stated as the ‘contextual prior’. The estimate is obtained after the posterior probability exceeds a threshold. Let us restate Eq.the decision making,

$$\begin{gathered} \widehat{x}=\text{arg}\underset{x}{\text{max}}p(x|{\epsilon }^{S_{0:C}},{\mathbf{\Phi }}^{S_0:C}) \\ \text{subject to:}p(x|{\epsilon }^{S_{0:C}},{\mathbf{\Phi }}^{S_{0:C}})\ge \tau \end{gathered}$$ (3)

Here, C is the smallest possible integer. The optimization problem in Eq.(3) tells us that, the framework keeps presenting sequences unless posterior distribution of any of the symbol passes a threshold value τ. We can observe that the decision making procedure consists independent sequence updates as denoted in Eq.(1). In this equation we need sequence likelihoods to update the posterior. In next section we investigate the sequence update.

Posterior Update in sequence

For simplicity in this section we assume the sequence of interest is S_i. Also for simplicity we can assume the prior probability in a sequence S_i is the posterior $p\left(x|{\epsilon }^{S_{0:i-1}},{\text{Φ}}^{S_{0:i-1}}\right)$. This directly follows the independence of sequences. In the sequence update assuming $p\left({\epsilon }^{S_i}|{\mathbf{\Phi }}^{S_i}\right)$ in Eq.(1) is a common factor we are only interested in $p\left({\epsilon }^{S_i}|x,{\mathbf{\Phi }}^{S_i}\right)$ for the particular sequence,

$$p(x|{\epsilon }^{S_{0:i}},{\mathbf{\Phi }}^{S_{0:i}})\propto p(x|{\epsilon }^{S_{0:i-1}},{\mathbf{\Phi }}^{S_{0:i-1}})p\left({\epsilon }_{0:N}^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right)$$ (4)

here ε denotes the EEG samples obtained in the sequence and Φ denotes the flashed letters in order where Here M denotes the total number of the flashed symbols and N denotes the length of the discrete time sequence. We aim to come up with an update procedure for letters. Assume we have an alphabet X of size |X|. We want to update the posteriors $p(x|{\epsilon }_{0:N}^{S_0:i},{\mathbf{\Phi }}_{0:M}^{S_{0:i}})$ of each letter x ∈ X.

We presented in the model Fig. 1 the latent variable ℓ. It carries the information of each EEG sample throughout the sequence. ℓ_i ∈ {0, 1} where 1 means the EEG sample has ERP component, 0 means the EEG sample does not belong to the target symbol flash.

We know from RSVP Keyboard™, due to the inter stimuli interval, the EEG samples are correlated, so do the labels. This piece wise dependency is modeled in Fig. 1–(a). We further define the time labelling in 1-(b). Using such representation allows us to represent the temporal targetness for each time sample. Also defining such a label vector will allow us to exploit the sequential structure of the data. Unfortunately, it can be observed that defined labels does not become independent of each other as the dependency is caused by the intent x. This problem can be tackled with using label definition. In order to expand this definition to each EEG sample, we marginalize the latent variable over $p\left({\epsilon }^{S_i}|x,{\mathbf{\Phi }}^{S_i}\right)$ in Eq.(4). This leads us to,

$$\begin{gathered} p(x|{\epsilon }^{S_{0:i}},{\mathbf{\Phi }}^{S_{0:i}}) \\ \propto p(x|{\epsilon }^{S_{0:i}-1},{\mathbf{\Phi }}^{S_{0:i-1}})\sum _{L}p\left({\epsilon }_{0:N}^{S_i}|{\mathcal{l}}_{0:N}^{S_i}\right)p\left({\mathcal{l}}_{0:N}^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right) \\ \propto p(x|{\epsilon }^{S_{0:i-1}},{\mathbf{\Phi }}^{S_{0:i-1}})\sum _L\frac{p\left({\mathcal{l}}_{0:N}^{S_i}|{\epsilon }_{0:N}^{S_i}\right)}{p\left({\mathcal{l}}_{0:N}^{S_i}\right)}p\left({\mathcal{l}}_{0:N}^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right) \end{gathered}$$ (6)

Observe that in Eq.(6), the proportional term is obtained after applying Bayes’ rule over the EEG likelihood. Also notice that L denotes set of all possible ℓ combinations. This constant factor is $p\left({\epsilon }_{0:N}^{S_i}\right)$. We can easily say, the factor is common for each letter, therefore we easily ignore the effects of this term by proportionally estimating the posterior probabilities. We also have the freedom of using a discriminative classifier for the inference. To update the posterior we require the definitions of the terms in Eq.(6).

We can easily observe $p\left({\mathcal{l}}_{0:N}^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right)\in \{0,1\}$ which works as a selection operator. We can observe this relation from the definition of the latent variable,

$$\begin{gathered} p\left({\mathcal{l}}_{0:N}^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right)=\prod _{k=0}^{N}p\left({\mathcal{l}}_k^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right) \\ p\left({\mathcal{l}}_k^{S_i}|x,{\mathbf{\Phi }}_{0:M}^{S_i}\right)=\{\begin{array}{ll}1\hfill & \text{if}k\in \left[t_m,t_m+\text{fs}/2\right]\hfill \\ 0\hfill & \text{otherwise}\hfill \end{array} \end{gathered}$$ (5)

where fs denotes the sampling frequency of the system. Probability distribution of latent label variables $p\left({\mathcal{l}}_{0:N}^{S_i}\right)$ depend on the symbol display policy. Remember we denote the displayed symbols ${\mathbf{\Phi }}^{S_i}$ and the intended symbol x. We say that $p\left({\mathcal{l}}_{0:N}^{S_i}=0^{N\times 1}\right)=p\left(x\notin {\mathbf{\Phi }}_{0:M}^{S_i}\right)=(|X|-M)/M$. We also know the display policy randomly inserts the probable letters into the display list. Therefore each of the other possible labeling have the probability 1/|X|.

Conditional probability distribution over latent variables $p\left({\mathcal{l}}_{0:N}^{S_i}|{\epsilon }_{0:N}^{S_i}\right)$ can be rewritten using the model in Fig. 1,

$$p\left({\mathcal{l}}_{0:N}^{S_i}|{\epsilon }_{0:N}^{S_i}\right)=\prod _{m=0}^{N}p\left({\mathcal{l}}_m^{S_i}|{\mathcal{l}}_{<m}^{S_i},{\epsilon }_m^{S_i}\right)$$ (6)

where < m denotes all possible indices less than m. This is possible as in a sequence S_i labels are not independent of each other as the influence interval of ${\mathbf{\Phi }}^{S_i}$ are overlapping. This allows us to treat the label as the latent state. Finding an estimate for state conditional probabilities in Eq.(6) can be achieved using long short term memory (LSTM) based recurrent neural networks (RNNs) [13, 14, 15]. Observe that RNN’s are also used to factoring graphical models [16] similar to our application. RNN as a function approximate can easily output state probabilities throughout the time series by exploiting the sequential structure. We can train the network by minimizing Kullback Liebler divergence. Once we get a tight approximate of the probability distribution we can insert everything into Eq.(6) to calculate the probability distribution over symbols.

3. EXPERIMENTS / RESULTS

Here in this section we question the performance of the proposed model by comparing it with the current framework in the RSVP Keyboard™. Brain signals are recorded in 16 channels separately using a 10–20 EEG measuring system with channels ‘Fp1, Fp2, F3, F4, Fz, Fc1, Fc2, Cz, P1, P2, C1, C2, Cp3, Cp4, P5, P6’. To increase the signal quality, signals are drifted using a FIR linear-phase bandpass filter in [1.5, 42][Hz] with zero DC gain and a notch filter at 60[Hz]. This data is used for determining the user intent.

Instead of using 16 channel data, we focus 4 channels C1, C2, Cz and Fz. We observed that empirically, increasing number of channels, increases the number of parameters to be estimated dramatically, which results in bad performance in temporal training procedure also decreasing number of channels actually reduces the complexity of the system as well. We also observed that incorporating frontal channels Fp1, Fp2 makes the model confused and increases discrimination difficulty of ERP from eye gaze potentials. We also give details about the steps in the method of training the neural network, in which we used the Chainer environment [17].

We train the recurrent neural network with fully connected 4 LSTM layers with 35 nodes which is justified through cross validation. We minimized KL divergence using a conjugate gradient descent method. We demonstrate a cross validation of a particular architecture in Fig. 2. We can observe, training error converges to zero, however there are some validation sets that does not match with the rest of the data. Observe that, we plot only one training error as it has a similar pattern in other validations.

Fig. 2:

Training error and validation error samples in a training. We can observe, training data is not explicitly labeled as it is extremely user dependent. In the data there exists data without response that damages the training procedure (Validation error does not decrease at all).

As we discussed in earlier sections, declaring a latent variable for each time stamp allows us to visually give a distribution over event related potential presence. We give the temporal plot in Fig. 3. We can observe from the figure, the temporal plot of probability distributions of latent variables follows the relationship with Eq.(6), also learning a temporal model allows us to locate the ERP response in the observed EEG. The second plot in Fig. 3 gives us an intuition, depending on the framework, the user might have performed weak or responded to another symbol which is also captured with this model.

Fig. 3:

Correct detection and misclassification example. We can observe that the RNN is capable of giving a probability distribution over all latent variables. In the first example ERP is localized. In the second example we can observe that based on the training procedure the RNN captures some ERP shaped responses where the labeled are not considered to be ERP.

We compared the method using calibration data from 8 different users. We trained an RNN with 10 fold cross validation on data and choose the model with minimum validation error. We calculated the AUC scores using the same procedure as conventional RSVP does and visualize the scores in Fig. 3. We can observe that classifying using sequence data increases the performance for badly classified trials. We observed also there are users, whose ERP can be distinguished from non-ERP using a quadratic boundary. Therefore we conclude by stating, taking the entire sequence into consideration increases the performance in poorly classified data sets.

4. CONCLUSION

In this paper, we proposed a graphical model for a sequence that incorporates temporal dependency. We observed proposed model leverages poorly classified users. This is promising as the users with AUC lower than 0.75 values could not manage to type accurately in online sessions. With proposed method we can increase the typing performance of individual users and help them communicate. We also denote that hyper parameters in conventional model are optimized explicitly for AUC (separability of classes), instead the proposed model is trained to detect ERP activity. We can state that proposed model performs similar/better using a comparison metric where the conventional model has the strong hand. As described before, we want to investigate the typing speed improvement with the method.

We observed from experiments sequential modeling degrades the performance for high AUC users with values around 0.9. This can be explained with the Gaussianity of feature vectors of particular users. We aim to get an ensemble method with both models to provide individual user’s ability to type without decreasing other user’s performance. In our future work we want to try different architectures on RNN to avoid such errors. We also want to incorporate the probabilistic nature of ERP appearance. As we observed through experiments, the users can react to non-target symbols or can not react to targets. Therefore we can include a confusion matrix which denotes the probability of response appearance.

Fig. 4:

We calculate the AUC values for sequence posterior update, using posterior multiplicative scores as target or non-target scores. Conventional RSVP calculates scores for target and non-target trials.