Hierarchical Graphical Models for Context-Aware Hybrid Brain-Machine Interfaces

Abstract

We present a novel hierarchical graphical model based context-aware hybrid brain-machine interface (hBMI) using probabilistic fusion of electroencephalographic (EEG) and electromyographic (EMG) activities. Based on experimental data collected during stationary executions and subsequent imageries of five different hand gestures with both limbs, we demonstrate feasibility of the proposed hBMI system through within session and online across sessions classification analyses. Furthermore, we investigate the context-aware extent of the model by a simulated probabilistic approach and highlight potential implications of our work in the field of neurophysiologically-driven robotic hand prosthetics.

I. Introduction

Non-invasive brain interfaces have shown the promise of providing alternative communication and control means for people with neuromuscular disabilities. In the context of neurophysiologically-driven hand prosthetics, hybrid brain-machine interfaces (hBMIs) based on fusion of electroencephalographic (EEG) and electromyographic (EMG) activities to decode upper limb movements gained significant interest [1], [2]. In that regard, to investigate neural correlates of human motor behavior, a variety of recent studies have shown promising results in both EEG-based [3–8] and EMG-based [9–11] settings for decoding of complex same hand gestures. In the light of recent promising work, we argue that probabilistic fusion of multimodal information sources in a unified framework would yield significant insights to develop robust hybrid BMIs for neural prosthetics.

In this pilot study, we present a novel hierarchical graphical model based context-aware hybrid brain-machine interface to decode human hand gestures. As suggested by previous studies on the human hand grasping movements taxonomy [12], we utilize a hierarchical scheme of hand gestures. Based on experimental data collected from three healthy subjects during stationary posture executions and subsequent imageries of five different hand gestures with both limbs, we demonstrate feasibility of the hBMI system through both within session and online across sessions classification analyses. We utilize probabilistic fusion of multimodal neurophysiological information sources with respective state-of-the-art feature extraction protocols in EEG and EMG signal processing. Furthermore, we investigate the context-aware extent of the model by a simulated probabilistic approach and highlight potential implications in the field of neurally-controlled robotic hand prosthetics for amputees.

II. Methods

A. Participants and Data Acquisition

Three healthy subjects (one male, two female; mean age 27.7 ± 3.1) participated in this study. Two of the subjects were right handed. Before the experiments, all participants gave their informed consent after the experimental procedure was explained to them in accordance with guidelines set by the Northeastern University Institutional Review Board.

During the experiments, two g.USBamp biosignal amplifiers (g.tec medical engineering GmbH, Austria) were used for data recordings. EEG data were recorded at 19-channels with 1200 Hz sampling frequency, using active EEG electrodes placed on the scalp according to the international 10 − 20 system at locations F3, F4, FC5, FC3, FCz, FC4, FC6, C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CPz, CP4 and CP6. EMG data were recorded from six bipolar electrodes with 1200 Hz sampling frequency, located on the extensor digitorum, flexor carpi ulnaris, flexor digitorum superficialis, extensor carpi ulnaris, brachioradialis and pronator teres muscle groups of both arms (c.f. [11] for an illustration).

B. Study Design

Experiments involved stationary executions and concurrent motor imageries of different hand gestures [13], [14]. All participants attended the experiment for five sessions on separate days of a week. Flow of the experiments in each session were the same. An experiment session consisted of eight blocks of 50 trials each, resulting in a total of 400 trials per session. Each trial in a block corresponded to stationary execution of one of five different hand gestures for five seconds with brief intermissions of three seconds between trials. Visual cues were presented to the participants indicating the specific hand gesture for the corresponding trial (see Figure 1). Participants were instructed to preserve a stationary hand posture representing the gesture, and perform concurrent motor imagination of the corresponding grasping(i.e. medium wrap, power sphere, parallel extension or palmar pinch) or resting (i.e. open palm) action at each trial. Order of the gestures across trials were randomized. Participants performed the gestures with the right hand in four of the blocks, and with the left hand in the other blocks with alternating order starting with the dominant hand.

Fig. 1.

Hierarchical flow of the gestures in the same hand [12], with corresponding visual cues presented during experiments for right hand gestures. Horizontally flipped images were displayed for left hand gestures.

C. EEG Signal Processing Pipeline

EEG data were pre-processed by downsampling to 300 Hz and performing baseline correction on each trial by subtracting the time-series mean activity of the first second, from the subsequent 1 to 5 second interval. Resulting four second trials were split into 250 ms windows as the data point samples to be used for further analyses. Feature extraction is performed separately between the classes at each disjunction of the hierarchical chart for binary decision making.

Common spatial patterns (CSP) algorithm and its filter bank CSP extensions (FBCSP) are state-of-the-art methods used for spatial filtering of EEG signals in the context of BCIs [15], [16]. Feature transformation with CSP aims to identify a discriminative basis for a multichannel signal recorded under different conditions, in which signal representations maximally differ in variance between these conditions. In a binary paradigm, CSP algorithm solves:

$$w^{*}=\underset{w}{ \text{argmax} }\left\{\frac{w^T{\Sigma }_{X|c_1}w}{w^T{\Sigma }_{X|c_2}w}\right\}$$

where ${\Sigma }_{X|c_1}$ and ${\Sigma }_{X|c_2}$ denote the class covariance matrices of the data matrix X for classes c₁ and c₂. Obtained CSP filters ω are then used for data matrix transformation.

To extract FBCSP features at each hierarchical disjunction, pre-processed EEG data were separately filtered using fourth order Butterworth bandpass filters on two motor movement associated frequency bands α-band (8 – 15 Hz) and β-band (15 − 30 Hz) [15]. From both frequency bands, 6 discriminative CSP filters were used in feature extraction [4]. As a result, 12-dimensional EEG feature vectors at each hierarchical disjunction were constructed with log-normalized variances in FBCSP transformed components.

D. EMG Signal Processing Pipeline

EMG data were filtered using a 60 Hz notch filter, and a fourth order Butterworth bandpass filter between 20 – 500 Hz. Without baseline correction, time-series windowing into 250 ms intervals was performed over the four second time interval starting after the first second until the end of the trials. Raw time-series features were obtained by calculating the root-mean-squared (RMS) values in each one of the six EMG recording sources in every time window [11].

Primarily based on the notion of a lower-dimensional muscle synergy framework controlling a wider range of hand movements [17], low-rank representation of multichannel EMG activity through non-negative matrix factorization (NMF) provides significant insights [9–11]. In particular, as a non-negative least squares problem, NMF algorithm solves:

$$\begin{gathered} \underset{H,W}{ \text{minimize} }\Vert V-WH{\Vert }_F^2 \\ \text{ subject to }H\ge 0\text{ and }W\ge 0 \end{gathered}$$

where $V\in {ℝ}^{N_C\times N_T}$ denotes the data at N_T time samples, $W\in {ℝ}^{N_C\times N_S}$ is the base matrix as a transformation between the number of EMG recording sources N_C and the number of synergies N_S in the low-rank representation, and $H\in {ℝ}^{N_S\times N_T}$ is the activation matrix.

Extracted six-dimensional RMS vectors from the time windows are concatenated to generate matrix V, which is used to calculate the base matrix on model training data. Obtained base matrix is then used for extracting activation levels H as features. Number of synergies N_S for the base matrix is chosen as 5. Hence, binary decision making at each disjunction of the hierarchical chart was performed using 5-dimensional EMG feature vectors constructed as activation levels of each base component.

E. Context-Aware Bayesian Decision Making

Proposed hybrid BMI system is illustrated through the graphical model in Figure 2. For any trial t;

Fig. 2.

Graphical model of the hBMI system at trial t. Observed random variables are indicated with blue nodes: ϵ_t represents EEG features, ξ_t represents EMG features, and C_t represents external context information. Final decision ℓ_t is deterministically related (dashed lines) with the states at each level. Observed random variables are probabilistically related with the states at each level (solid lines).

• ${\mathit{ϵ}}_t^{(i)}$ denotes the EEG feature vector at level i,

• ${\xi }_t^{(i)}$ denotes the EMG feature vector at level i,

• C_t denotes the external context information,

• $S_t^{(i)}$ denotes the state at level i,

• ℓ_t denotes the class label of the current trial.

Decision making across hand gestures at any trial t is performed by maximum-a-posteriori (MAP) estimation over the class labels ℓ_t, using both EEG, EMG and context evidence through the objective function:

$${\widehat{\mathcal{l}}}_t=\underset{{\mathcal{l}}_t}{\text{argmax}}P\left({\mathcal{l}}_t|{\mathit{ϵ}}_t^{(0)},{\mathit{ϵ}}_t^{(1)},{\xi }_t^{(1)},{\xi }_t^{(2)},{\xi }_t^{(3)},C_t\right)$$

Estimated class label ℓ_t is deterministically related with the combination of individual decisions at each one of the four levels. Assuming equal priors for the observed random variables and using the independency assumptions on the random variables inherent through the graphical model, we obtain the following decision criterion:

$$\begin{gathered} =\underset{{\mathcal{l}}_t}{\text{argmax}}\{P\left({\mathit{ϵ}}_t^{(0)}|S_t^{(0)}\right)P\left({\mathit{ϵ}}_t^{(1)}|S_t^{(1)}\right)P\left({\mathit{\xi }}_t^{(1)}|S_t^{(1)}\right) \\ P\left({\xi }_t^{(2)}|S_t^{(2)}\right)P\left({\xi }_t^{(3)}|S_t^{(3)}\right)P\left(S_t^{(3)}|S_t^{(2)},C_t\right) \\ P\left(S_t^{(2)}|S_t^{(1)},C_t\right)P\left(S_t^{(1)}|S_t^{(0)},C_t\right)P\left(S_t^{(0)}|C_t\right)\} \end{gathered}$$

where the expressions are denoted as follows:

• $P\left({\mathit{ϵ}}_t^{(i)}|S_t^{(i)}\right)$: EEG likelihood probability at level i estimated from training data,

• $P\left({\mathit{\xi }}_t^{(i)}|S_t^{(i)}\right)$: EMG likelihood probability at level i estimated from training data,

• $P\left(S_t^{(i)}|S_t^{(i-1)},C_t\right)$: State transition probability given the previous state i − 1 and context information.

III. Analysis and results

Decision making is performed in 250 ms time windows. At each step, MAP estimation over class labels is performed with the obtained decision criterion. To evaluate the likelihoods for features at each level, multivariate Gaussian kernel density estimations are performed with the training data.

A. Decoding Using Neural Information

To perform decoding solely based on neural information, state transition probabilities based on context information priors are assumed to be uniform. Classification analyses are performed through both within session 5 × 5-fold cross-validation protocols, as well as an online decoding framework using data from all previous sessions (i.e. days) in model training to test for the subsequent day, for Sessions 4 and 5. This online protocol is equivalent to using zero neural training data to operate the system on testing session day.

Proposed hBMI graphical model performs 10-class decoding with five gestures from both hands (see Table I). Also, performance based only on the extracted EEG and EMG features are tested. Specifically, 4-class decoding solely based on the EEG features at the first two levels (S⁽⁰⁾ and S⁽¹⁾) of the hierarchical chart, and 5-class decoding based only on EMG features from the last three levels (S⁽¹⁾, S⁽²⁾, and S⁽³⁾) of the right hand gestures junction are performed. Chance levels for 4-, 5-, and 10-class decoding problems are 25%, 20%, and 10% respectively.

TABLE I.

Classification accuracies of within session and online decoding analyses for all subjects. Cross-validated accuracies are averaged across five repetitions. 5-class EMG classifications are presented only for right hand gestures.

		5-Fold Cross-Validation Within Sessions						Online Performance
		Session 1	Session 2	Session 3	Session 4	Session 5	Mean	Session 4	Session 5	Mean
Subject 1	EEG - 4 Class	44.2%	38.6%	42.2%	48.0%	47.7%	44.1%	34.6%	32.4%	33.5%
	EMG - 5 Class	92.6%	89.5%	92.3%	89.9%	94.5%	91.7%	72.4%	54.9%	63.6%
	hBMI - 10 Class	76.3%	65.5%	76.3%	86.5%	86.3%	78.1%	52.1%	46.3%	49.2%
Subject 2	EEG - 4 Class	40.5%	32.0%	30.7%	33.5%	34.3%	34.2%	30.2%	29.2%	29.7%
	EMG - 5 Class	79.5%	84.5%	81.9%	89.3%	85.9%	84.2%	36.0%	20.2%	28.1%
	hBMI - 10 Class	57.6%	51.0%	51.1%	57.8%	55.5%	54.6%	18.3%	15.0%	16.6%
Subject 3	EEG - 4 Class	42.9%	38.1%	36.1%	35.4%	35.5%	37.6%	24.1%	24.5%	24.3%
	EMG - 5 Class	67.2%	70.7%	80.0%	78.3%	79.8%	75.2%	44.1%	48.9%	46.5%
	hBMI - 10 Class	53.5%	55.0%	52.3%	53.0%	57.6%	54.2%	20.9%	20.3%	20.6%

Results in Table I demonstrate average classification accuracies up to 44% for EEG-based 4-class decoding between resting and grasping with both hands, 91% for EMG-based 5-class right hand gesture decoding, and 78% for 10-class decoding with the hBMI system in within session analyses. Highest online performance of the hBMI system using at least three days of training data is obtained as a 49% average accuracy by Subject 1, above the 10% chance level.

B. Embedding External Context Information

We performed simulated analyses by introducing arbitrary external context information probabilities at each one of the four hierarchical levels. With an intuitive example, this could be interpreted as an independent probabilistic output from a vision based classification algorithm by a camera mounted on the neurally controlled prosthetic hand. In that case, detection of an object in sight would increase probability of grasping at S⁽¹⁾, or binary decoding based on the object type to determine a power versus precision grasp would affect at S⁽²⁾ [18]. Results in Table II demonstrate that a simulated context information probability with a value in range 0.75 to 0.60 provided any level can increase 10-class online decoding accuracies of the hBMI model up to 54%.

TABLE II.

Online 10-class Session 5 decoding accuracies of the hBMI model with simulated context information probabilities.

	Subject 1	Subject 2	Subject 3
$P\left(S_t^{(0)}|C_t\right)$ = 0.75	54.1%	32.1%	31.8%
$P\left(S_t^{(1)}|S_t^{(0)},C_t\right)$ = 0.70	46.4%	15.4%	22.1%
$P\left(S_t^{(2)}|S_t^{(1)},C_t\right)$ = 0.65	47.8%	16.1%	23.7%
$P\left(S_t^{(3)}|S_t^{(2)},C_t\right)$ = 0.60	47.6%	15.9%	23.5%

IV. Discussion

In the present study, we demonstrate feasibility of a novel graphical model based context-aware hBMI system for hierarchical hand gesture decoding. Proposed model incorporates EEG and EMG evidences in a unified framework, where we predefine which individual neural information source is incorporated for decoding at each hierarchical level. We highlight that the probabilistic concept using hierarchical pairwise decision making with multimodal information sources provides significant insights for the field of neurally-controlled robotic hand prosthetics for amputees. Beyond its demonstrated validity, lack of across sessions transfer learning approaches is likely to confound online model performance as observed through the naive implementation of online analyses. This is potentially a result of high variability in daily neural recording qualities, as well as the inherent non-stationarity property of the neural sources.

In future work, beyond considering day-to-day model learning, feature extraction protocols for the neural sources will be expanded for consistency with the independence assumptions directed by the graphical model. In that regard, utilizing hierarchically adversarial feature extraction schemes is expected to provide better performances. Finally, going beyond the primary motor cortex, recent work on broader EEG correlates of motor learning [19], as well as of reach-to-grasp movements [5–7] is likely to provide better insights on the EEG features that can be exploited.