Asynchronous Decoding of Grasp Aperture from Human ECoG During a Reach-to-Grasp Task

Abstract

Recent studies in primate neurophysiology have focused on decoding multi-joint kinematics from single unit and local field potential recordings. However, the extent to which these results can be generalized to human subjects is not known. We have recorded simultaneous electrocorticographic (ECoG) and hand kinematics in a human subject performing reach-grasp-hold of objects varying in shape and size. All Spectral features in various gamma bands (30–50 Hz, 70–100 Hz and 100–150 Hz frequency bands) were able to predict the time course of grasp aperture with high correlation (max r = 0.80) using as few as one ECoG feature from a single electrode (max r for single feature = 0.75) in single trials without prior knowledge of task timing. These results suggest that the population activity captured with ECoG contains information about coordinated finger movements that potentially can be exploited to control advanced upper limb neuroprosthetics.

I. Introduction

It has been shown that neurons in primary motor cortex (M1) code for various grasp movements [1], as well as object and grasp types [2]. Recent studies in the field of brain-machine interfaces (BMI) have shown that kinematic-tuned single- and multi-unit activity from neuronal populations in M1 can be used to control a cursor in a human subject with tetraplegia [3], and to decode individual finger movements [4] or up to 25 upper limb joints in monkeys [5] with model prediction accuracies as high as 0.73 (Pearson’s r). Local field potential (LFP) recordings have also been used to decode grasp movements in primates with high accuracy [6].

Electrocorticography (ECoG) is emerging as a potential neural interface modality for BMI systems due to its high bandwidth compared to electroencephalography (EEG) [7] and its potential for greater stability than spike recordings [8]. ECoG is uniquely positioned between the microscale of animal recordings and the macroscale of EEG. Relative to microelectrode-recorded LFPs, ECoG records the activity of larger populations of neurons, and the smaller distance between electrodes relative to EEG and cortical sources gives it greater sensitivity to high frequency activity.

Recently, it has been shown that ECoG activity in the high gamma range (>60 Hz) contains information about the timing and amplitude of periodic flexion/extensions of individual fingers [9]. In addition, the smoothed instantaneous amplitude of ECoG, known as the local motor potential (LMP), has also been demonstrated to contain information for cortical decoding of individual finger flexion and extension [9] and slow grasping motions [10] during periodic movements.

In this study, we investigate the feasibility of using modulations in ECoG spectral and time-domain features to decode coordinated finger movements during non-periodic and behaviorally relevant movements of the upper limb. ECoG signals were recorded from a human subject while he reached towards and grasped a battery of objects of varying shapes and sizes. Specifically, we examined how each ECoG feature could predict the time course of grasp aperture.

II. Methods

A. Experimental Paradigm

The subject was instructed to reach towards and grasp different objects. The subject was seated comfortably with his arm in a neutral position. Reach-and-grasp trials were performed in blocks consisting of 15 to 20 trials. All objects were presented in each block in a pseudo-random sequence. The subject performed 4 blocks of the experiment during the recording session, which lasted approximately 30 minutes. Upon the experimenter’s verbal cue (“Start”), the subject reached out and grasped each object with his fingertips. The subject was not instructed to pick up the objects. The objects were placed approximately 40 cm from the subject’s resting hand position and aligned to the center of the body. Three sets of objects were used, 1) rectangular blocks with a height of 14 cm and a width (w) of 3, 5, 6.5, 8, 9.5 or 11 cm and, 2) circles with a diameter (d) of 4, 7, 10, 13, or 16 cm, and (3) trapezoids with a height of 14 cm, a taper angle (θ) of 15 or 30 degrees, and a shorter base width (b) of 1 or 5 cm (Fig. 1A). The trapezoidal objects were presented separately with the shorter base both on the top and on the bottom. After a hold period varying between 2.5 and 6.5 sec, the subject was verbally cued (“End”) to return his hand to its resting position. The inter-trial interval was varied randomly from 5–10 seconds. The subject was given 2–3 minute breaks between each block of the experiment. The trial timeline is illustrated in Fig. 1B.

Fig. 1.

A. Three sets of objects of parametrically varying shapes and sizes were presented to the subject. B. Timing of the experimental trial. C. Electrode placement map. The electrodes are darkened where upper extremity movements were elicited or interrupted with ESM. An encephalomalacic lesion of ventral sensorimotor cortex is shaded orange.

B. Subject characteristics and ECoG recordings

Our subject for this paper was a 12 year old male with medically refractory simple partial seizures with focal sensory, then motor, manifestations in the left upper extremity. Neonatal seizures with left motor manifestations had occurred 8 hours after birth but after medications were weaned at 3 months, they did not recur until 10 years of age. A CT had revealed a blood clot in the right parietal lobe shortly after birth, and MRI now revealed ventral peri-central encephalomalacia. Although the patient initially had had a weak left grasp, now there were no focal motor or sensory deficits on exam, and he reported being left-handed. The patient was temporarily implanted with subdural ECoG electrode grids for localization of the ictal onset zone and electrocortical stimulation mapping of eloquent cortex to guide surgical treatment. The location and duration of electrode implantation was determined solely by clinical need. ECoG grids (Adtech, Racine, WI) consisted of 10–64 platinum disc electrodes (diameter 4 mm, 2.3 mm exposed, 10 mm center-to-center spacing), embedded in a soft Silastic sheet and implanted on the cortical pial surface. A 128-channel EEG amplifier (Stellate Systems, Montreal) with 300-Hz low-pass filter was used to acquire 71 ECoG channels (with respect to an intracranial reference electrode) at 1 kHz sampling rate. Simultaneously, joint angles from the hand contralateral to the implantation site were recorded using a 22-sensor CyberGlove (Immersion Corp, San Jose, CA) with a 25 Hz sampling rate. One sensor between the thumb and index finger best represented grasp aperture. Electrodes were localized by co-registration of highres pre-implant MRI with post-implant CT using anatomical fiducials. Results from routine electrocortical stimulation mapping (ESM) are shown in Fig. 1C. Electrodes where ESM elicited motor responses are marked with black dots.

C. Time-frequency Feature Extraction

Feature extraction was performed using the Signal Processing Toolbox in MATLAB (MathWorks, Natick, MA). After re-referencing ECoG signals with a common average reference (CAR), spectral power was extracted from 256 ms windows shifted every 40 ms (CyberGlove sampling period) using the Burg algorithm for autoregressive spectral estimation with model order 25. Spectral power was first log transformed and then averaged across five frequency ranges: mu, 8–14 Hz; beta, 16–30 Hz; low gamma, 30–50 Hz; high gamma 1, 70–100 Hz, and high gamma 2, 100–150 Hz. A local motor potential (LMP) amplitude feature [11] was also extracted by taking the average of a 500 ms window prior to each kinematic sample.

D. Model Construction and Evaluation

For each frequency band, a generalized linear model (GLM) of grasp aperture during an entire experimental block was constructed using an individual feature from one channel (i.e., channel i) at multiple latencies.

$$y(t)={\beta }_{i,0}+\sum _{\tau =-800ms}^{0ms}{\beta }_{i,\tau }·x_i(t-\tau )+{\epsilon }_i$$ (1)

where y(t) is the model-predicted grasp aperture, x_i(t−τ) is the feature at τ ms latency relative to time t, and β are coefficients of the model. Latencies between −800ms to 0ms in 40ms time steps were included in the model. Models were constructed and evaluated under five-fold cross validation. Pearson’s r correlation was then calculated between observed and predicted grasp aperture across the concatenated testing sets.

A pooled GLM was also constructed using all the latencies from all the channels in individual frequency bands, as well as in a combination of all bands, which were added in descending performance order. Performance in this context was evaluated in a nested validation loop, where single feature models were sequentially constructed on 80% of the training set and evaluated for the remaining 20%. The specific performance criterion for inclusion in the pooled model was Pearson’s r computed between the observed and predicted grasp aperture in the training set only. The pooled model can be described as follows:

$$y(t)={\beta }_0+\sum _{i=1}^N\sum _{\tau =-800ms}^{0ms}{\beta }_{i,\tau }·x_i(t-\tau )+{\epsilon }_{\mathit{pooled}}$$ (2)

with notation as defined above. Since we observed that the r-values saturated with fewer than 20 electrodes, only models with N varying from 1 to 20 are shown for this subject. Pooled models were constructed (i.e., including feature selection) and evaluated under five-fold cross validation.

For both single feature and pooled models, the five-fold cross validation was repeated twenty times using random testing and training segmentations to reduce the effects of segmentation-specific accuracy estimates.

III. Results

A. Single Feature Decoding Accuracy

Models constructed from single features with multiple lags showed high decoding accuracies with both high gamma features, with mean Pearson correlation coefficients (r) as high as 0.74 (70–100 Hz) and 0.75 (100–150 Hz). The best performing electrodes for these high gamma features were concentrated in sensorimotor regions of cortex (Fig. 2). The peak correlation between actual and predicted grasp aperture for single electrodes for LMP, mu, beta, and low gamma were −0.04, 0.24, 0.50, and 0.61, respectively. The percentage of ECoG channels that achieved statistically significant correlation (p < 0.05, Bonferroni corrected) varied greatly between feature types: LMP = 0.0%, mu = 4.2%, beta = 12.7%, low gamma = 38%, high gamma 1 = 31.0%, high gamma 2 = 26.8%. The low gamma band in aggregate had the most electrodes that were significant across all blocks and permutations, but had lower peak electrode correlations than the high gamma features. Generally, only features at 0ms and −800ms latencies had significant weights (p < 0.05, Bonferroni corrected) in the GLM of grasp aperture.

Fig. 2.

Spatial distribution of correlations between actual and reconstructed grasp aperture for each frequency band. The electrodes with highest correlation in the 30–50 Hz, 70–100 and 100–150 Hz ranges were localized over sensorimotor cortex.

B. Pooled Feature Decoding Accuracy

Models were constructed with varying numbers of channels (i.e., N = 1 to 20), with features either restricted to one particular frequency band (or LMP) or drawn from the entire population of features. Pooled feature model performance (Pearson’s r) was averaged across all the permutations of all four blocks, and the significance of the correlation was calculated. All models drawn from the entire population of features were significantly correlated for all tested values of N (p < 0.05, Bonferroni corrected) (Fig. 3A). All models drawn only from individual spectral feature types (e.g., mu features only) also were significantly correlated for all tested values of N. No amount of pooling of LMP features could cause model predictions to become positively correlated with observed joint angle traces. Models constructed from the two high gamma features outperformed the other spectral features, as expected from the single feature decoding results. Table I summarizes the correlation between observed and predicted grasp apertures for each of the feature types and select N. For all the frequency bands, only a few channels were needed to achieve the maximum correlation with the least number of channels for 70–100 Hz and 100–150 Hz models (4, 10, 6, 3, and 3 channels for Mu, beta, low gamma, high gamma 1, and high gamma 2, respectively).

Fig. 3.

A. Correlation between predicted and actual grasp apertures as a function of the number of channels in the decoding model using ECoG power in each frequency band (and pooled from all frequency bands). Highest correlations could be achieved using as few as three ECoG channels. B. Actual and reconstructed grasp aperture joint angles for each of the four recording sessions (GLM N = 20). The mean observed grasp aperture was subtracted from both the observed and predicted traces for display purposes only.

TABLE I.

Pooled Feature Decoding Performance Summary

Feature Type	N = 1	N = 5	N = 10	N = 15	N = 20
LMP	−0.04	−0.10	−0.16	−0.21	−0.25
mu	0.26	0.39	0.42	0.44	0.44
beta	0.50	0.59	0.60	0.61	0.61
low gamma	0.62	0.71	0.71	0.72	0.72
high gamma 1	0.74	0.77	0.78	0.78	0.78
high gamma 2	0.75	0.76	0.78	0.78	0.78
All	0.75	0.78	0.79	0.80	0.80

Fig. 3B shows the traces of actual and predicted grasp apertures using all the features during four different sections of the experiment. Both the transients as well as the stationary values of grasp aperture can be predicted with high accuracy (for all plots, 0.79 < r < 0.81).

IV. Discussion

Here, we have developed models predicting hand kinematics using multiple spectral features and latencies for a human subject performing reach-to-grasp movements. We have shown that human ECoG signals can predict grasp aperture with high correlations in single trials of behaviorally relevant, non-periodic movements. ECoG spectral features in the 70–100 and 100–150 Hz bands were highly tuned to grasp aperture. Furthermore, these features were localized to sensorimotor cortex and utilized a range of times before each predicted time point. Surprisingly, the LMP features which have been shown before to correlate with periodic opening and closing of the hand [10] did not appear to generalize to the non-periodic movements we studied.

In our subject, ECoG spectral features recorded from a small number of electrodes were sufficient to achieve the highest correlations between observed and predicted kinematics. These results strengthen previously published evidence that ECoG signals can be utilized for future high degree of freedom BMI systems [9, 11].