Impact of Time-Frequency Representation to the Generalization Ability of Synthesized Time-Frequency Spatial Patterns Algorithm in Brain Computer Interface

Abstract

This paper focuses on the problem of how time-frequency representation influences the generalization ability of the ‘synthesized time-frequency spatial pattern (TFSP)’ algorithm in Brain Computer Interface (BCI) for classification. TFSP methods use time-frequency analysis to extract features in both time and frequency domains. Different time-frequency analysis methods have been used before. However, it is still unknown how these different approaches influence the generalization ability. We compared the performance of three different TFSP methods in classifying 3 stroke survivors’ intention in hand opening and closing. Each of these TFSP methods uses different time-frequency analysis approaches with different time-frequency resolutions. Our results show that a high resolution in time-frequency resolution doesn’t guarantee better generalization ability. It seems that although large redundancy in feature reduces the generalization ability of TFSP method, certain redundancy is necessary for achieving high generalization ability.

I. INTRODUCTION

THE synthesized time-frequency spatial patterns (TFSP) algorithm was proposed by Wang et al. [1]. Basically, this method is consisted by two parts: the time-frequency analysis and the classification. The time-frequency analysis represents the original signal in time-frequency domain by coefficients in different time-frequency grids. Each of these grids then determines the class of the input signal using a certain classification algorithm. Finally a synthesis scheme is used to make a final classification based on the determination results from all the time-frequency grids.

To improve the overall performance of this method, one can improve either feature extraction or the classifier. Recently, we have enhanced the performance of the original TFSP method by replacing the original linear nearest neighborhood method with a support vector machine [2, 3]. Using this enhanced method, we have reported average classification accuracy of 92% over four healthy subjects and 74% over two stroke subjects for classifying a static shoulder or elbow torque generation. As for the time-frequency feature extraction, the original method proposed by Wang uses a band-pass filter bank with 50% overlap in frequency domain combined with the averaged signal energy in a moving time window [1]. More recently, Qin and He reported a wavelet-based TFSP approach for classifying motor imagery tasks [4]. However, no comparison between the performance of band-pass filter-based TFSP and wavelet-based TFSP has been reported. Therefore, it is still unknown the possible impact of time-frequency analysis to the performance of TFSP algorithm in Brain computer interface.

In general, techniques of signal representation using time-frequency analysis project the time-domain signal to a set of kernel functions and thus representing the original signal by linear combinations of kernel functions and appropriate coefficients. For example, short time Fourier Transform represents the original signal by a set of the windowed trigonometric functions with appropriate coefficients. These coefficients can fully represent the original signal in a combined time-frequency domain, and provide detailed information about what and when a frequency component occurs. The size of time and frequency window that is used to analyze the signal determines the time-frequency resolution. For example, a kernel function that has a small size time-window will give a high resolution in the time domain, because a small size time window can tell the occurrence of an event more accurately than a large time window. Due to Heisenberg uncertainty principle, one cannot achieve high resolution in both time and frequency domain at the same time. The time-frequency resolution will determine the sparsity, or redundancy, of the resulted time-frequency representation. When the resolution is high, a sparse representation of the original signal can be achieved and thus reduce the redundancy. Specifically for the TFSP method, a sparse representation will result in a more independent classification in each of the time-frequency grids. On the contrary, a low time-frequency resolution will result a dense representation with high information redundancy.

In many cases, one believes that the reduction of redundancy can help to improve the generalization ability of classifiers [5]. However, there are also cases showing that a sparse representation of the signal reduce the generalization ability of a classifier [6]. Therefore, it is still unknown how the time-frequency representation will impact the overall performance of TFSP method. In this paper, we compared generalization ability of TFSP method based on representations of EEG signal using three different time-frequency analysis approaches: the original band-pass filter (BPF) bank based, continues wavelet transform (CWT) based, and bionic wavelet transform (BWT) based approaches. Both CWT and BWT use a Morlet kernel function.

Band-pass filter bank and CWT have been well documented in the previous literature. Therefore, we only briefly mentioned the concept of BWT here. BWT is a special wavelet transform which provides an additional adaptation between time and frequency resolutions [7]. A key difference between CWT and BWT is that although the time-frequency window is fixed at a particular frequency band in CWT, window size in BWT adapts to the change of the amplitude of the signal in that frequency band. For example, to process a 20 Hz signal lasting from 0 to 100 ms then switching to 10 Hz at 100 ms, a window with higher time resolution at 100 ms will be used. BWT has been proved to be able to provide better trade-off between time and frequency resolution [7]. In this paper, we have particularly chosen to use a CWT that provides a higher time-frequency resolution than the BPF, and a BWT that has the same product of time-frequency resolutions while with better trade-off between them. More specified information of time-frequency analysis was stated in the method section below.

II. METHODS

A. Experiment

We tested the performance of TFSP methods using the high-density (160-channel) surface EEG signal collected during 3 stroke subjects performing either a hand opening or a hand closing task after reaching a target and then holding the hand posture for 2 s. These 3 moderately to severely impaired chronic hemiparetic stroke subjects (1 male and 2 female) have either no or very limited voluntary control of the impaired hand. All subjects provided written consent that was approved by the Institutional Review Board of Northwestern University, prior to participation in this study.

During the experiment, surface EEG, EOG and 2-channel of EMG signals from the wrist extensor and flexor were sampled at 1 KHz and stored in the computer for further processing.

B. Signal preprocessing

The recorded 160-channel EEG signals were preprocessed using Brain Vision Analyzer V1.04 software (Brain Products GmbH, München, Germany). Using this software, we first manually removed the noisy channels, bad intervals of eye artifacts and drifting signals. The earlier onset of two EMG signals was used to segment the EEG signals (1 s before and 1 s after EMG onset). The remaining data were then baseline corrected and down sampled to 256 Hz. In order to reduce smearing effects on EEG measurements due to the head volume conductor, a spatial high-pass filter, i.e., finite difference surface Laplacian (SL) Error! Reference source not found., was applied to the signal from each of the channels before exporting the data for further analysis. Furthermore, peripheral electrodes were removed and preprocessed EEG signals from the remaining inner electrodes <131 were exported for BCI classification using TFSP method.

C. Time-frequency synthesized spatial patterns algorithm

A flow chart for the data processing of TFSP method has been given in [2]. Basically, a time-frequency analysis was used to extract the event related desynchronization information for the EEG signal from each of the channels. This time-frequency analysis was applied to the EEG signal from every channel and thus created a spatial pattern in each of the time-frequency grids. These spatial patterns, one grid by another, were then sent to a nearest neighborhood classifier that makes a decision based on which standard spatial pattern gives the closest distance to the input spatial pattern. Finally, a simple ‘voting’ synthesis process was used: if the majority of these time-frequency grids determine the input is a hand-opening task, then the final determination is an opening task; otherwise, it is a hand closing task. Three different TFSP methods were investigated. The only difference among them is the time-frequency analysis method. These methods are as described below.

Method 1 --- Band-pass filter bank based method

We repeated the exact method as proposed in the original TFSP method by Wang [1]. Briefly, EEG signals ranging from 5 to 34 Hz were first decomposed into a series of frequency bands using a group of constant Q value (Q = 4) band-pass filters. Each of the filters is overlapped with the center frequency of the previous one becoming the starting frequency of the consecutive one. Subsequently, the Hilbert transform was used to extract profiles of the oscillatory activities, and the resulting profiles were then divided into equal-length intervals (55 ms of each interval) with a 50% overlap. By dividing signals in both frequency and time domains, EEG signal were presented in time–frequency grids. In our study, the time–frequency grids consist of 13 frequency bins and 72 time segments. Following the divisions, we calculated the instantaneous power (i.e., integration of the profile) in each of the time–frequency grids, and thus, a time-frequency representation was obtained by coefficients in these 72×13 time-frequency grids for signal from one EEG channel. In each grid, the time-frequency analysis was repeated in every EEG channel, and thus generating a spatial pattern in each of these time–frequency grids.

Method 2 --- Continues wavelet transform

A continues wavelet transform (CWT) with Morlet function was used to analysis the signal. Totally, 13 scales that cover center frequencies from 5 to 36 Hz were applied to EEG data from each of the EEG channels. (Center frequencies from scale number 1 to 13 are 36.5639, 31.2675, 26.7384, 22.8653, 19.5532, 16.7209, 14.2988, 12.2276, 10.4564, 8.9418, 7.6465, 6.5389 and 5.5917 Hz.)

Method 3 --- Bionic wavelet transform

BWT coefficients were calculated based on corresponding WT coefficients [8]. The two active factors G₁ and G₂ for calculating the BWT [7, 8] were set as G₁=0.001, G₂=0.015.*log (n+3), where n is the number of the scale. And the saturation constant BWT_s was set as 0.9. Once again, the CWT or BWT was applied to EEG signal channel by channel, and thus generating a spatial pattern in each of these time–frequency grids.

D. Measure of generalization

We measure the performance of a TFSP method in terms of generalization that is the ability of this method to correctly classify never seen before data. Leave-four-out (LFO) accuracy rate was used to provide a practical way to estimate the generalization ability of a method given a finite number of training data [9]. Let us consider the data set S={(xi, yi)}, i=1 to N. For each i, remove four samples {(xi, yi), (x_i+1, y_i+1),(x_i+2, y_i+2),(x_i+3, y_i+3)} from S to be the testing set and the remaining samples compose the training set. If among the four testing samples, n (0≤n≤4) of them was classified correctly, the accuracy rate, AC_i, is then equal to n/4. After cross-validateing the above process for I times (I=20 in this study), the overall LFO accuracy rate is then given by $\frac{1}{I}*\sum \left(A{C}_i\right)$.

III. RESULTS

The time-frequency patterns of a signal trial EEG signal from the channel Cz in subject 1 are plotted in figure 1. In figure 1, the representations were obtained using BPF, CWT and BWT-based methods, respectively. The x-axis shows the time bins from 1 to 72 covering from −1 s to 1 s with 0 representing the onset of EMG signal; and the y-axis shows the scale number (for CWT and BWT) or the number of band-pass filters (for BPF-based method), where numbers 1 and 13 correspond to the highest (~35 Hz) and lowest (~5 Hz) center frequencies, respectively. In this figure, an example area was pointed out to show that BPF, CWT and BWT provide time-frequency resolution from low to high.

Fig 1.

Time-frequency representation obtained by BPF- (top), CWT- (middle) and BWT-based (bottom) methods. The x-axis represents the time windows covering from −1 s to 1 s (left to right) with 0 representing the onset of EMG. The y-axis represents the scale number corresponding to center frequencies from about 35 Hz to 5 Hz (bottom to top). The arrow pointed area shows an example of time-frequency representation obtained by the three methods that provide different time and frequency resolutions.

The averaged training accuracy rates in each of the time-frequency grids over 20 times of cross-validation were plotted in figure 2. We can see that for all these methods the alpha band provides higher accuracy rates during movement preparation and the static hand force generating phases. The beta band provides higher accuracy rates during the hand movement phase (a short window after movement onset corresponding number 36 time bin). From this figure, one also observed that results obtained by BWT have a more narrowly distributed high rate in the time window corresponding to the hand movement phase (time-bin 36 and scale 5). Furthermore, BWT resulted a more consistently distributed high accuracy rates in alpha band (scales 10–12) when compared to results obtained by BPF or CWT. However, overall accuracy rates obtained by BWT are lower than those obtained by BPF- or CWT-based methods in each of the time-frequency grids.

Fig 2.

The distribution of the mean test accuracy rate over 20 times of cross-validation from subject S1 in each of the time-frequency grids. Results were obtained using BPF- (top), CWT- (middle) and BWT- (bottom) based methods, respectively.

The LFO accuracy rate and the standard error for each of these TFSP methods were plotted in figure 3. A paired t- test showed that there is a significant (p<0.1) difference in the generalization ability among these methods.

Fig 3.

The LFO accuracy rates and the standard errors for BPF-, CWT- and BTW-based methods.

IV. DISCUSSIONS

We compared the generalization ability of BPF-, CWT- and BWT-based TFSP methods in classification stroke subject’s intention in opening or closing their paretic hand. The highest performance was obtained by CWT-based TFSP method. This method provides a better time-frequency resolution than BPF-based method. This improvement provided positive evidence showing improved time-frequency resolution (i.e., reduced redundancy) can improve the generalization ability of TFSP method. However, the usage of BWT, which has the same the product of time-frequency resolutions as CWT and better trade-off between them, resulted in the worst generalization ability of TFSP method. From figure 2, it seems that such decreased performance is caused by the reduction in accuracy rate achieved in each of the time-frequency grids. This result implies that a reduced redundancy may improve the generalization ability by improving the synthesizing scheme because each of the time-frequency can make a more independent decision. However, further reduced redundancy may decrease the classification ability for each of the time-frequency grids. It is worthy to note that BWT creates ripples in time-frequency representation. The non-smooth time-frequency representation of BWT can be another reason for the deduction of the classification ability of each of the grids. Furthermore, the statistical analysis is based on data collected on 3 subjects. Based on this small number of subjects, we obtained significant results with the achieved power equal to 84% (BPF-CWT), 100% (CWT-BWT), and 86% (BPF-BWT). Further investigation using more thoroughly designed time-frequency analysis (i.e., with gradually increased time-frequency resolutions) methods and based on larger number of subjects is required for a clear answer of this question.