High Accuracy Decoding of User Intentions Using EEG to Control a Lower-Body Exoskeleton

Abstract

Brain-Machine Interface (BMI) systems allow users to control external mechanical systems using their thoughts. Commonly used in literature are invasive techniques to acquire brain signals and decode user’s attempted motions to drive these systems (e.g. a robotic manipulator). In this work we use a lower-body exoskeleton and measure the users brain activity using non-invasive electroencephalography (EEG). The main focus of this study is to decode a paraplegic subject’s motion intentions and provide him with the ability of walking with a lower-body exoskeleton accordingly. We present our novel method of decoding with high offline evaluation accuracies (around 98%), our closed loop implementation structure with considerably short on-site training time (around 38 sec), and preliminary results from the real-time closed loop implementation (NeuroRex) with a paraplegic test subject.

I. INTRODUCTION

Having the potential of increasing the quality of life for paraplegic and tetraplegic population, BMI systems to control lower-body and upper-body exoskeletons became a focus of research for the past decade. Researchers recently reported the control of external physical devices (robotic manipulators, prostheses) and computer cursors using invasive methods [1], [2], [3]. Some major limitations of invasive methods are the risks associated with surgery and degradation in signal quality over time. Non-invasive methods typically acquire brain signals using scalp electroencephalography (EEG). Although having a small signal-to noise ratio compared to the intracortical methods, recent results show the possibility of using non-invasive (risk-free) decoding of delta-band brain activity using EEG to predict the human limb movements to reliably drive a BMI [4], [5], [6]. We have also reported the feasibility of a single session training followed by a successful on-line decoding [7] of EEG signals.

Parametric methods such as Kalman Filter, Wiener Filter [4], [8] and soft computing methodologies [9] are used to accomplish high decoding accuracies of limb motion parameters, such as the joint angles and joint velocities. In this paper, we direct our attention to a shared control architecture, that benefits from the control of definitive parameters of walking by the exoskeleton system. There are a number of systems in the literature designed to assist or provide walking for stroke patients, spinal cord injured and elderly population such as the Human Assistive Limb Exoskeleton (HAL, Cyberdyne Inc.) and the ReWalk (Bionics Research Inc.). In our studies we are using the Robotic Exoskeleton (REX, REX Bionics Ltd.). One advantage of this system is that it is self balancing and it has a variety of preprogrammed motions, the most important for our studies being the walking, turning, sitting, standing motions. Having the advantage of such a system performing an inherent closed-loop control once a motion command is received, we examined the possibility of decoding the intended user motion rather than the motion parameters (fig. 1). This requires a classifier approach of decoding compared to the mapping of EEG signals to continuous set of joint parameters. We present our novel decoding model architecture and the offline decoding results for repeated walking-turning right-turning left motions and sit-rest-stand motions. We also present our initial findings on an EEG-based BMI system to control the REX (NeuroRex) exoskeleton in a real-time closed-loop setting, resulting in independent walking for the paraplegic user.

Fig. 1.

A paraplegic volunteer controlling NeuroRex with his thoughts via our proposed EEG BMI interface

II. Materials and Methods

A. Experimental Protocol

The Institutional Review Board of the University of Houston approved the experimental protocols. After giving informed consent, subject conducted two tasks. Although the REX system can be controlled via a joystick by the user, in Task 1 subject is asked to follow and complete a path marked on the ground while exoskeleton is controlled by an operator remotely. The rationale behind this task is to have the user focus only on the given walking-turning right-turning left task while minimizing the effect of the subject’s hand/finger motions on the data set. The path is a discrete number 8 figure where each linear sections’ connection angles are compatible with the robot’s right/left turn angles. Task 2 was conducted with the remote control interface for repeated sit-rest-stand-rest cycles for 5 minutes.

B. Data Acquisition and Pre-Processing

A 64 Channel electrode cap (actiCAP, Brain Products GmbH) was placed on the head of the subject according to the international 10–20 system having FCz as reference and AFz as ground. A wireless interface (MOVE system, Brain Products GmbH) was used to transmit data (sampled at 100Hz) to the host PC. Data then filtered in the (0.1–2Hz) range using a 2nd order Butterworth filter. Filtered data were then standardized (z-score). Separate channels were then used to create a feature matrix (to extract the EEG delta-band amplitude modulation information) using a 200ms window with 1 shift, each row having the time shifted from [ch₁ (t−n), …, ch₁ (t), ch₂ (t−n), …, ch₂ (t)…ch_m (t)], where n is the window size in samples and m = 64 is the total number of channels.

C. Classification Method for Decoding

Having such a feature matrix structure often yields a high dimensional set (for all 64 channels used, the set is 1280 dimensional). Commonly used dimensionality reduction techniques such as Principal Component Analysis (PCA) or Fisher’s Linear Discriminant Analysis (LDA) work under the assumption that the distributions are Gaussian, whereas real-life observations are often non-gaussian and in some cases are multimodal [10]. As proposed in [11], we use a classification paradigm that was designed to preserve and use the rich statistical structure of the data. We use a Local Fisher’s Discriminant Analysis (LFDA) to reduce the dimensionality of the data while preserving the multimodal structure, and employ a Gaussian Mixture Model (GMM) classifier to map the states of the exoskeleton to the feature matrix (amplitude modulation of the subject’s delta-band brain activity).

1) LFDA

For a data set with samples $\mathbf{X}={\{x_i\}}_{i=1}^n$ in ℝ^d (d is the dimension of the feature space, n is the total number of samples), and class labels (in our case: the motion states) y_i ∈ {1, 2, …, c} (c is the number of classes), the affinity (heat kernel) between x_i and x_j is defined as

$$A_{i,j}=\mathit{\text{exp}}(-\frac{{\Vert x_i-x_j\Vert }^2}{{\gamma }_i{\gamma }_j})$$ (1)

which measures the distance among data samples. Here ${\gamma }_i=\Vert x_i-x_i^{(knn)}\Vert$ denotes the local scaling of data samples in the neighborhood of x_i, and $x_i^{(knn)}$ is the k_nn nearest neighbor of x_i. In LFDA, the local between class and within class scatter matrices (S^lb and S^lw) are defined as

$$S^{lb}=\frac{1}{2}\sum _{i,j=1}^n{W}_{i,j}^{lb}(x_i-x_j){(x_i-x_j)}^T$$ (2)

$$S^{lw}=\frac{1}{2}\sum _{i,j=1}^n{W}_{i,j}^{lw}(x_i-x_j){(x_i-x_j)}^T.$$ (3)

W^lb and W^lw are,

$$W_{i,j}^{lb}=\{\begin{array}{cc}{A}_{i,j}(1/n-1/n_l),\hfill & \mathrm{i}\mathrm{f}{y}_i=y_j=l\hfill \\ 1/n,\hfill & \mathrm{i}\mathrm{f}{y}_i\ne y_j\hfill \end{array}$$

$$W_{i,j}^{lw}=\{\begin{array}{cc}{A}_{i,j}/n_l,\hfill & \mathrm{i}\mathrm{f}{y}_i=y_j=l\hfill \\ 0,\hfill & \mathrm{i}\mathrm{f}{y}_i\ne y_j\hfill \end{array}$$

where n_l is the number of training samples for the lth class. Thus the transformation matrix (T_LFDA) to reduce the dimensionality of the feature space is defines as

$$T_{LFDA}=\text{arg}\underset{T_{LFDA}}{\text{max}}\mathrm{t}\mathrm{r}[{\mathrm{\Psi }}_{lw}^{-1}{\mathrm{\Psi }}_{lb}]$$ (4)

where ${\Psi }_{lw}=T_{LFDA}^T{S}^{lw}{T}_{LFDA},{\Psi }_{lb}=T_{LFDA}^T{S}^{lb}{T}_{LFDA}$ and S^lbT_LFDA = ΛS^lwT_LFDA for diagonal eigenvalue matrix Λ. Defined by the offline analysis, this transformation matrix was used to reduce the dimensionality of the feature matrix formed in real-time.

2) GMM

A gaussian mixture model is a combination of two or more normal gaussian distributions [12]. A typical GMM probability density function is defined as

$$p(\mathbf{x})=\sum _{k=1}^K{\alpha }_{k}N(\mathbf{x},{\mu }_k,{\mathrm{\Sigma }}_k)$$ (5)

where $N(\mathbf{x},{\mu }_k,{\mathrm{\Sigma }}_k)=\mathrm{\Gamma }\mathit{\text{exp}}(-\frac{1}{2}{(\mathbf{x}-{\mu }_k)}^T{\mathrm{\Sigma }}_k^{-1}(\mathbf{x}-{\mu }_k))$ for $\mathrm{\Gamma }=\frac{1}{{(2\pi )}^{d/2}|{\mathrm{\Sigma }}_k{|}^{1/2}}$. The mixing weight α_k, mean µ_k and covariance matrix Σ_k are estimated by the expectation-maximization algorithm [13]. As for the transformation matrix for the dimensionality reduction, these parameters were identified in the offline analysis and used to estimate, in real-time, the probabilities of a given feature vector belonging to one of the classes. A more detailed discussion of the LFDA-GMM method can be found in [11].

III. RESULTS

We report the offline analysis results for training our GMM distribution for two different cases (tasks 1 and 2), where the subject was asked to attempt (in his mind) the walk, turn-right, turn-left motions and sit-down, rest, stand-up motions. Online implementation results reported in this section discusses the preliminary results of our closed loop control implementation for walking or stopping with the exoskeleton.

A. Open-loop analysis

Our EEG decoding method to identify the user intent provided a good performance for a range of reduced dimensionality and k-nearest-neighbor (knn) parameters.

After the pre-processing step, we grouped the feature matrix data of different classes (3 classes for the tasks reported). We then randomly selected 1500 samples for training, and from the remaining data, random 1500 samples for testing, from each class. This corresponds to 9.7% and 16.2% of the total data length for tasks 1 and 2 respectively. Figure 2 shows the validation accuracy surface for varying dimension and knn values (1 to 451 for both). The accuracy for this case is sensitive to the reduced dimension, but less sensitive to the knn value. The mean accuracy change from reduced dimension 51 to 81 for all knn values is an increase of 12.93%, whereas a knn change from 1 to 81 (where the maximum accuracy of 99.07% occurs) for all dimensions yields a mean accuracy increase of 1.39%. Using fixed reduced dimension and knn parameters which yields high accuracies, the whole data set were evaluated using the corresponding GMM distribution. This validation step was designed to be a pseudo real-time implementation. Namely, the specified windows size of data were read from the pre-processed data and the probabilities of the feature vector belonging to each class were calculated, identifying the class label as the one with maximum probability. Figure 3 shows this test of evaluating the entire data set using the LFDA-GMM method. The overall accuracy associated with this case is 98.17%. These analyses were repeated for a second case study using the data set from task 2. Similarly, the accuracy surface over iterated knn and reduced dimension values are shown in figure 4. The associated pseudo real-time estimation is reported in figure 5. The overall accuracy associated with this test is 99.68%.

Fig. 2.

Accuracy surface for task 1 gridded for 451 reduced dimension and 451 knn values

Fig. 3.

Evaluation of the task 1 model using the entire data set

Fig. 4.

Accuracy surface for task 2 gridded for 451 reduced dimension and 451 knn values

Fig. 5.

Evaluation of the task 2 model using the entire data set

B. Closed-loop implementation

Our closed-loop implementation sessions consists of two steps. One for data acquisition, dimensionality reduction and GMM model training, and one for closing the loop in real-time and simultaneously recording the related EEG activity for future analysis. Having identified average values that yields low dimensions (around 80) and high accuracies (around 93%), our model training and offline validation to confirm the accuracy takes on average 45 seconds.

We developed a multi-threaded C++ code for the closed-loop implementation of our method, having a thread for the real-time EEG data acquisition, and another thread for the data preprocessing and GMM evaluation. The data acquisition thread continuously receives the EEG data and saves it in a data file. After receiving a specified window size of samples, the data matrix is passed to the second thread. The second thread is responsible for pre-processing the data by filtering it using a 2nd order Butterworth filter between [0.1 2 Hz] using the overlap-add technique, forming the feature vector, standardizing it and performing dimensionality reduction using the identified transformation matrix (eq. 4). This thread is also responsible for calculating the probabilities of the feature vector belonging to each class and identifying the class label as the one with maximum probability. It also converts the class label to the associated motion of the exoskeleton (stop or walk forward) and transmits it via the RS232 protocol.

As an initial study, we directed our focus on a relatively simpler task of walking or stopping with the exoskeleton. In a single session we observed an increased voluntary control of the exoskeleton by the subject after several trials with the same model. After performing four exploratory trials with the exoskeleton, the subject reported that he was able to stop the exoskeleton and walk again with it repetitively at least three times in a three minutes long trial. In our latest experiments we have asked the subject to stop the robot and stay standing for as long as he can, as we recorded our model’s output of walk or stop continuously. We conducted a total of 8 trials. Each trial lasted until a total of 5 stops were achieved. We then calculated the percentage of the stop signals transmitted to the exoskeleton from the total length of the recorded commands. For the first 4 trials we have recorded and increased the percentage of stops from 97.11% to 99.32%. Following a break of 45 minutes after the 4th trial, we recorded a monotonic decrease of 1.15%, followed by an increase of 1.52%, bringing the accuracy to 99.69% for the last trial. These results are indeed very promising, but yet to be conclusive. The reason for that is an inherent time delay of the exoskeleton. As described before, the exoskeleton was designed to be stable — it thus has to complete a whole cycle of gait before stopping, resulting in a slow time response compared to our model’s output. We have also timed the exoskeleton’s full stops manually and calculated the percentage of stops from the overall experiment duration. We have seen a monotonic increase of accuracy from 21% to 70% over the first period of trials, and a final accuracy of 90% for the last trial. Our next step is to account for this time delay thus increasing the exoskeleton’s response to the timed commands we record from our model’s output. It should be noted that, at the time of this writing, we have only run a total of four sessions with the subject and the increased number of correct responses motivates us towards an implementation including several tasks (walking, stopping, turning right and left, sitting and standing).

IV. DISCUSSION

Having the locality preserving dimensionality reduction combined with a multimodal GMM classification indeed performed very well for a range of control parameters. We also ran a series of offline cross validation tests to check the performance of our decoder method. We tested a single model in an iterative setting for 1500 test samples randomly selected for each of the 20 iterations. The mean accuracy observed from this test was 97.74 ± 1.2% for task 1 and 99.31 ± 0.54% for task 2. We have also tested our model generated for task 1 with the data from task 2 (each having 3 classes) as an additional control. We observed a mean accuracy of 18.78% for this case, and a mean accuracy of 8.9% for the case when we test the task 2 model with the task 1 data. This suggests that our models are extracting EEG features unique to the tasks for which they were originally trained for. We are now increasing the number of classes by training our models with a larger number of tasks (i.e., having multiple types of motions).

Using the proposed decoding methodology, we clearly see an increase of correct command executions over trials. In our closed-loop setting, we also have more successful trials when using an on-site trained model compared to a model that is trained in a previous session. That brings the importance of a more adaptive learning scheme (identifying and updating the parameters as the experiment is performed). It should also be noted that the overall control scheme encapsulates the subject’s adaptation to a given task, thus a longitudinal study investigating the control performance of a fixed model over sessions is also important. One of our future directions is to embed the identification of the reduced dimension and knn parameters, as well as building the GMM distribution model in real-time. Another important step is to account for the inherent time delays associated with the exoskeleton to increase the responsiveness to the commanded motions. As discussed previously, the exoskeleton performs a series of motion-cycles before executing a new command. In several trials we observed the transmission of a new command before the execution of the previously received command was completed. For example, before a stop command was fully executed, a new walk command would be received from the BMI algorithm, which explains the difference between the observed and recorded command percentages over the entire data set. Nevertheless, we see an increase in accuracy in both cases. We are currently continuing longitudinal testing with a small cohort of SCI patients, as well as able bodied control subjects.