Hand posture classification using electrocorticography signals in the gamma band over human sensorimotor brain areas

Abstract

Objective

Brain machine interface systems translate recorded neural signals into command signals for assistive technology. In individuals with upper limb amputation or cervical spinal cord injury, the restoration of a useful hand grasp could significantly improve daily function. We sought to determine if electrocorticographic (ECoG) signals contain sufficient information to select among multiple hand postures for a prosthetic hand, orthotic, or functional electrical stimulation system.

Approach

We recorded ECoG signals from subdural macro- and microelectrodes implanted in motor areas of three participants who were undergoing inpatient monitoring for diagnosis and treatment of intractable epilepsy. Participants performed 5 distinct isometric hand postures, as well as 4 distinct finger movements. Several control experiments were attempted in order to remove sensory information from the classification results. Online experiments were performed with 2 participants.

Main Results

Classification rates were 68%, 84%, and 81% for correct identification of 5 isometric hand postures offline. Using 3 potential controls for removing sensory signals, error rates were approximately doubled on average (2.1×). A similar increase in errors (2.6×) was noted when the participant was asked to make simultaneous wrist movements along with the hand postures. In online experiments, fist versus rest was successfully classified on 97% of trials; the classification output drove a prosthetic hand. Online classification performance for a larger number of hand postures remained above chance, but substantially below offline performance. In addition, the long integration windows used would preclude the use of decoded signals for control of a BCI system.

Significance

These results suggest that ECoG is a plausible source of command signals for prosthetic grasp selection. Overall, avenues remain for improvement through better electrode designs and placement, better participant training, and characterization of non-stationarities such that ECoG could be a viable signal source for grasp control for amputees or individuals with paralysis.

1. Introduction

Brain machine interface (BMI) systems translate electrical activity from the nervous system into command signals for assistive technology [1, 2]. This has the potential to provide treatment for a wide variety of neurological conditions. One particular application of interest is generating command signals for finger and grasp movements. Such a system would be enabling for the large population of individuals who can move their arms to control their hand position, but cannot position their fingers or form a grasp. This includes a small population of hand amputees (~1000/yr in the US, [3]) and a larger population of spinal cord injury patients with cervical level injuries between C5–C7. According to one survey [4], there are 1.3 million people in the US suffering from paralysis from a spinal cord injury, approximately a third of which may be at the C5–C7 level [5]. Myoelectric (EMG) based control of prosthetic hands, the current state of the art for hand amputees, can allow users to produce a variety of hand postures. These control signals cannot be readily obtained in spinal cord injury patients. Even among amputees, ~30–40% do not regularly use their prostheses [6, 7]. In one survey, “limited usefulness” was given as the primary reason for non-use [6]. These studies suggest that better control signals are needed for prosthetic limbs and other assistive technology for the hand, including orthotic devices and functional electrical stimulation [8].

BMIs may have the potential to provide accurate and intuitive hand posture control directly from neural signals. Several neural signal sources are available. EEG, obtained from non-invasive electrodes placed on the scalp, has been used to generate control signals [9–13] which have been used to control prosthetic hand grasp [14–17]. Currently, multi-state classification performance of these systems is below performance demonstrated by myoelectric control [18, 19]. Also, the patient population is high functioning, usually employed [6, 7], and less likely to wear scalp electrode systems throughout the day.

Implantable systems based on short, penetrating microelectrodes in cortex have demonstrated accurate offline classification of finger movements in monkeys, achieving >99% classification of 12 movements, flexion and extension of the five fingers and wrist [20–22]. However, the use of subdural or epidural surface potentials may have advantages over current penetrating electrode systems. For example, a larger area is associated with finger movements in humans than monkeys. Although neural activity related to individual fingers in cortex tends to be mixed and overlapping [23–26], the somatotopy present could motivate a larger coverage area. Currently, only field potentials can be acquired across multiple square cm from a single implant.

Previous BMI work using ECoG signals falls into two major categories. The first is the use of indirect signals with robust modulation to drive different movement commands [27–30]. For example, Leuthardt and colleagues [27] used tongue movements and speech generation to move a cursor in 2 dimensions. This type of system may hold some promise for providing a means of communication for people with “locked in” syndrome from brainstem stroke or neuromuscular diseases such as ALS, or for basic study of neural plasticity. However, current commercial prosthetic hand control systems are often unused because the control signals are non-intuitive. Therefore, it would be ideal to use existing signals in motor cortex that are already associated with finger movements and grasps. Several studies using movement related brain activity have focused on fairly stereotyped motions [31–33]. This makes it difficult to determine if the decoded signal contains target information, or information about individual arm and hand components, since a signal that detects many aspects of movement could be linearly regressed against the movement to generate a positive correlation.

Much of the existing ECoG literature addressing native neural motor signals, associated with non-stereotyped movements, has focused on the hand and fingers [34–36], though one study has focused on larger arm movements [37]. Finger movements may work particularly well because finger representations are more spread out in cortex than, for example, upper arm muscles, which control endpoint [38]. Because ECoG relies on the summed activity of a population of neurons, behavioral tuning is highly dependent on the amount of somatotopy present. I.e., a substantial population of neurons beneath the recording electrodes must have similar properties. With this requirement, the larger brain area dedicated to fingers may present an advantage. One study has noted that spatially distinct brain regions may be specific to particular fingers [35], enabling robust prosthetic control signals. These data were collected while participants performed tapping motions with individual fingers. From these data, 1 of 5 fingers could be classified with an average of 80% correct (20% chance) [34]. A similar study using smaller Micro-ECoG electrodes achieved 73% decoding 1 of 5 fingers [39]. Finally, one study has shown successful offline classification of one of two grasps (power grasp and pinch) [36].

The present study supports and extends these results by considering a total of nine movement types, including four finger movements, four hand postures, and rest. In addition to classifying these movements, a number of controls to remove the effect of sensory information on decoder performance were tested. We also examined the robustness of these control signals in the presence of extraneous arm movement, and also across time. Finally, we present preliminary real-time online control of a prosthetic hand performing grasping movements. Routes to improvements remain, such as better electrode targeting and participant training.

2. Methods

2.1. Behavioral Tasks

Six participants, all undergoing invasive electrocorticographic monitoring for seizure localization, performed the tasks described in this study. However, only three participants had substantial sensorimotor coverage. The analyses presented here use a total of 18 10-minute datasets from these three participants (6 datasets each). Participants wore a dataglove (5DT) pictured in Fig 1a, while performing 9 different isometric hand motions. The glove provided finger flexion data as shown in Fig 1b. A sand-filled latex balloon was taped in the middle of the participants hand to provide isometric resistance to movement. The 9 movements, shown in Figure 1c, included 4 finger movements (flexion of the thumb, index finger, middle finger, and the ring and pinkie fingers together). Four types of isometric hand postures were also performed (a fist, pinch, point, and five finger splay). Movement instruction alternated with rest in 4 second halves of an 8 second trial. In addition to the initial rest period, the patient was also explicitly instructed to rest on some trials during the movement period. Participants were also instructed to rest the fingers not actively involved in the movement. They were cued by a laptop displaying photographs of either a resting hand or a hand in the final position of one of the 8 movements. During the experiment, an instructor demonstrated all movements for the patient, and performed the movements with the patient on every trial to encourage consistent behavior. Tasks were performed in several 10 minute blocks per session, including 70–80 trials each.

Figure 1.

(a) Picture of the data glove (5DT) used for measuring finger position. (b) Example data traces from the glove during an isometric thumb press trial. (c) Photographs that were used to cue eight movements (thumb, index, middle, ring, fist, pinch, point, splay and rest) (d) Signal processing example from one thumb trial on a well modulated electrode. From top to bottom, traces show the raw voltage, the referenced voltage, the data filtered with an elliptic (IIR) bandpass filter, between 66–114 Hz, and the log squared data smoothed with a 100 ms moving average.

Several variations of the isometric task were performed. In one block, participants were asked to imagine movements rather than performing them. In another block, participants were asked to perform simultaneous and alternating wrist flexion and extension while performing the finger and hand postures to determine the robustness of the neural decode. Wrist motion occurred throughout initiation of the hand posture, for approximately one second, and then the participant held the hand posture with the wrist flexed or extended for the remaining seconds. An instructor demonstrated these movements, encouraged the subject to practice for several trials, and continued to provide feedback to encourage consistency. Additionally, since most previous literature was done with tapping motions, we asked one participant to perform these motions for one block for comparison purposes. In the other two participants, a sensory experiment was performed where the instructor pressed on the fingertips with a pen for 4 second intervals, for approximately 10 trials each. No multi-finger hand postures were included in the sensory only datasets.

The task was sequenced using a real-time xPC (Mathworks) based platform with a 1 ms guaranteed execution time. Photographs of hand movements were displayed using a custom Java program running on a laptop, which sat on the participant’s bedside table. Commands were delivered to that program via UDP packets from the xPC.

2.2. Electrophysiology Recordings

Participants were implanted with a varying number of ECoG electrodes as shown in Figure 4a, according to clinical needs. Participants P1, P2, and P3 were implanted with macro electrodes, which were 4 mm in diameter and 1 cm apart as part of the normal clinical monitoring system. In each case, we recorded from the 64 electrodes with most relevant spatial coverage, generally sensorimotor areas. These were recorded through a splitter box that enabled simultaneous clinical monitoring throughout the experiments. In P2, 64 microwires (38 micron diameter) were embedded between the clinical macroelectrodes in groups of 2×2 electrodes 4mm apart. Microwires were also present for P3, but signals were highly corrupted by noise, and therefore not analyzed.

Figure 4.

(a) MRI images from P1, P2, and P3. Microelectrodes are shown as small red dots for P2. Images are color coded to denote the brain areas that the electrodes were classified into. (b) Performance classifying one of four hand postures and rest (chance = 20%) using only electrodes from a particular brain area, which is denoted by color. Only participant P1 had mesial electrodes, and only P2 had microelectrodes.

The placement of the electrodes with respect to central sulcus is shown in Figure 4a. Postoperative head computed tomography (CT) images were acquired from each subject after their implantation, which were subsequently aligned to preoperative structural T1 weighted magnetic resonance imaging (MRI) whole brain scans for localization and visualization of electrode location. Before CT-MRI alignment, MRI data were reoriented to AC-PC space by manually identifying the anterior commissure (AC), the posterior commissure (PC), and a third point in the midsagittal plane. MRI data were then resampled to 1mm isotropic voxels using a b-spline image interpolation algorithm from SPM5 (http://www.fil.ion.ucl.ac.uk/spm). Postoperative CT images, which clearly resolved the location of implanted electrodes, were then aligned to the T1 weighted MRI images using a mutual information algorithm, also implemented in SPM5. After CT-MRI alignment, electrodes were identified in the coregistered CT image slices and their centroid coordinates recorded. Because of postoperative shifts in brain position relative to the cranium, electrode coordinates were accordingly adjusted based on a local cortical surface projection as previously described [40]. Cortical surfaces were extracted via manual segmentation of the T1 MRI image using ITKGray a segmentation tool based on ITKSnap, and the Stanford mrVista package (http://white.stanford.edu/software). Electrodes were then visualized on each subject’s 3D rendered cortical surface as shown in Figure 4a using this cortical segmentation. By acquiring high resolution structural imaging for each subject, we could construct accurate 3D visualizations of electrode locations relative to each subject’s head space, accurate to within a few millimeters ( < 5 mm) of error [40]. This localization allowed for accurate anatomical classification of electrodes relative to clear gyral and sulcal landmarks for each subject. The accuracy of reconstructed electrode locations was validated by intraoperative photography and electrical brain mapping logs.

Up to 128 channels of neural data were recorded simultaneously using a Cerebus system (Blackrock Microsystems, Salt Lake City, UT). Data were stored at 30 kilosamples per second. These neural data were streamed into the xPC such that they could be filtered and decoded in real-time. This enabled neural data to be saved into the same data structure as the behavioral data without the need for subsequent time alignment.

2.3. Neural Data Analysis and Decoding

Neural data were recorded at 30 kilosamples per second, but down sampled to 1 kilosamples per second prior to analysis with the application of a boxcar filter. Common average referencing was performed across 32-channel blocks that shared the same physical cable, as this substantially reduced the remaining 60 Hz noise, as shown in Figure 1d. Several channels still had high noise, presumably due to poor contact with the brain. Channels were eliminated if they exhibited frequent amplifier saturation. Also, if the power of the “line noise”, which was the signal filtered to include only 60 Hz, was greater on average than twice the power of the signal overall, the channel was also excluded. Using these criteria, 234 out of 256 channels were included in the analyses presented here. All electrodes were included as features unless noted otherwise.

From each channel, power in the gamma band was obtained using an elliptic (IIR) band pass filter between 66 and 114 Hz, to avoid 60 Hz harmonics. The voltage data were then squared to obtain the power, and the log was taken to better approximate a normal distribution. Power in the beta band (10–30 Hz) in general was not used because it had substantially lower performance by itself, as described in the Results, and did not increase classification performance when added to the gamma band features. Lower frequencies approached the frequencies at which participants were performing arm movements. Therefore, we chose not to decode very low frequencies in case of movement artifacts.

For classifying movement types, a Naive Bayes decoder was used, similar to previous work [41], but using log gamma power as the feature set. Unless otherwise specified, for most decodes, the average log gamma power was calculated between 500 ms prior to movement onset to 1500 ms afterwards. Even in the case of isometric movements, a twitch could reliably be detected in the glove data to locate move onset. Isometric force continued for 4 sec, however, the bulk of power was located within these 2 second pieces. Log gamma power was assumed to have a normal distribution, and assumed to be uncorrelated between electrodes. The assumption of independence was appropriate for power in the gamma band, where 94% of the channels were correlated with one another with ρ<0.1, including the microelectrode data. This is consistent with several studies suggesting that the spatial extent of LFP at the cortical surface may be < 5mm [39, 42–45]. Lower frequency features, which we did not utilize, were more highly correlated. For each trial, the posterior probability of each movement was calculated, and the highest value was chosen as the result. Due to the low trial count, unless otherwise specified, the training and testing was done with single trial cross validation, so that for each trial, all other trials were used as the training set. All datasets included an equal number of trials for each movement type, including rest. Our performance metric is simply “percent correct”, which corresponds to the movements correctly classified, across all movement types and rest. Confusion matrices in Figure 3 provide more detailed information about the nature of the errors, but there was no systematic pattern across participants. Similarly “error rate” refers to the number of trials classified incorrectly over the total evaluated, and is equal to 1 - percent correct.

Figure 3.

Confusion matrices showing the percent correct when classifying each of four finger movements (thumb, index, middle, ring) and rest in the left most column, or each of five hand postures (rest, fist, pinch, point, and finger-splay) in the middle column. Each row corresponds to one participant. Percentage classified as a given hand posture is shown in the color bar. Perfect performance is a dark red diagonal line, and the overall percentage of correct answers is shown in the upper right. Rightmost column shows passive sensory finger touches in P2 and P3.

Online experiments were completed with participants P2 and P3 (and failed due to technical difficulties with participant P1). The log gamma power was decoded at a fixed time point 6.5 seconds into the trial, during the ongoing hand posture, based on the previous 2 seconds of data, though that integration window would be too long for use in a BCI system. The selected posture was output to a prosthetic hand, the iLimb Pulse (Touch Bionics, Livingston, UK), on the participant’s bedside table. All five fingers of this hand were controlled through an RS232 link, such that it could generate all the movements used in this study. Participants were asked to continue paying attention to the normal cue, to perform movements in the same way as they had during the training set, and to not pay attention to whether the signal was decoded correctly. However, the presence of the prosthetic limb did tend to draw the participant’s attention. The limb automatically reset to the resting state at the beginning of the next trial. Training data for the online decoder was generated from one ten minute block of approximately 70 movements, uniformly distributed between the movement types. For online experiments, in participant P2, a subset of 19 electrodes with the greatest power difference between the 9 movements and rest was chosen as the feature set. Some masking was also applied in the online experiment with P3, but the majority of the electrodes (49 out of 64) were included in the feature set. Fist versus rest was attempted online with both participants. P2 also attempted a 1-of-5 decode with rest and all four multi-finger hand postures. With P3, a 1 of 3 decode was attempted with both rest-fist-pinch and rest-fist-splay.

3. Results

3.1. Isometric Hand Posture Classification

Robust increases in gamma power were observed during hand movements for all participants on a subset of electrodes. All 3 participants had extensive sensorimotor coverage. Out of 234 low-noise electrodes, 41% had significantly different log power values between fist and rest (p < 0.05, t-test, Bonferroni corrected for 458 comparisons in this analysis). Less than a third as many or 11% had significantly different log power values between thumb movement and ring finger movement (p < 0.05, t-test, similarly corrected). Smaller percentages involving primarily the same electrodes differentiated other finger combinations. The large percentage differentiating fist from rest could include any electrode over a brain area involved in the task in any way, not just limited to movement generation. Figure 2a shows the gamma power from single trials during thumb and ring finger movement in participants P1 and P2. The electrodes shown are the best single electrodes outside of the primary sensory strip for differentiating these two movements. The overall “tuning” to hand movement can be seen in Figure 2b, where the average log gamma power is shown for each of the 9 hand movements.

Figure 2.

(a) Log power calculated from filtered voltage data in the high gamma band (66–114 Hz). Data shown from best single electrode (outside of primary sensory cortex) for differentiating thumb and ring finger movements. Single trial traces shown from the middle portions of the trials aligned to move-onset. Blue shows data from isometric thumb pressing trials, while green shows data from ring finger trials. Left panel shows data from participant P1 and right panel shows data from participant P2. Data were smoothed with a 2 second sliding window to reveal the trend (b) Average log power in the gamma band from these same electrodes during the 8 instructed movement types and rest, as labeled on the x-axis. Power was averaged across a 2 sec window starting 0.5 sec before movement onset, which was determined from glove sensor data. Error bars denote standard deviation.

To quantitatively interpret the amount of information included in these signals, neural data were classified offline using a Naive Bayes classifier as described in the Methods. Figure 3 shows confusion matrices that illustrate the success rate of these classifiers. In general, classifiers were successful in differentiating four fingers and rest (i.e. a 1 of 5 classification) during isometric pressing, with an average 79% correct (20% chance). Datasets included 35–40 trials each, with 7–8 repetitions of each of 5 conditions. This classification rate was higher than a single tapping dataset in P1, which demonstrated 65% correct. If only 500 ms was used instead of 2000 ms, starting at move onset, the percent correct went down to 70%. Comparing participants, P3 had extensive premotor coverage and also had a wide performance gap between fingers and hand postures. Also, P2 had extensive parietal lobe coverage and also had very high classification rates (100% correct) in the sensory task in which the participants’ fingers were passively touched with a pen.

When considering clinically viable hand prosthetics, hand posture selection may provide more utility than finger selection. The confusion matrices in Figure 3 shows classification of 1 of 5 types of hand postures, with an average 78% correct (20% chance). Rest, which is associated with lower gamma power, can be decoded quite accurately. Any single movement vs rest could be detected with an average 94% correct. Index finger flexion versus pinching could be considered similar movements, and could be decoded at 79% correct. Decoding of two hand postures versus rest would provide functionality comparable to current myoelectric prosthetic systems, and in this study could be classified at an average of 87% correct across all combinations of two hand postures and rest.

Decoding with log power in the beta band (10–30 Hz) had significantly lower performance than the gamma band (p < 0.01, t-test). While the beta band was useful in differentiating rest versus movement (90% correct, with 50% chance), it had approximately chance performance differentiating 1 of 4 fingers or 1 of 4 hand postures (28% correct, with 25% chance).

3.2. Controlling for Sensory Signals

Similar studies to date have shown comparable or better classification performance on finger movements. However, it is important to note that sensory signals are potentially abundant in ECoG recordings from sensorimotor areas, and will not be present in an amputee or spinal cord injury patient utilizing a prosthetic hand. Only one study to our knowledge has attempted to control for the fact that sensory signals may be inating these performance values by including only electrodes anterior to central sulcus [36]. We completed several analyses to evaluate the relative strength of sensory signals in the current dataset, and to explore various ways of removing their influence on decoder performance.

Figure 4a shows the MRI data from all three participants, with primary sensory cortex (S1) and primary motor cortex (M1) as well as parietal areas outlined in different colors. While the alignment precision (< 5 mm) may allow for several misclassifications, it is still informative to look at the trends. Electrodes were classified as being over motor (M1), sensory (S1), parietal, or mesial areas as shown. The number of electrodes in each brain area are shown in Table 1. Figure 4b shows the hand posture classification performance (1 of 4) using only electrodes from individual brain areas. Data are shown on three separate days from P1, and two days from participants P2 and P3, with macro- and microelectrodes shown separately on day 1 for participant P2. A subset of macroelectrodes over motor cortex were not recorded on day 2 due to connector error, so only microelectrodes are shown on day 2. While microelectodes and macroelectrodes showed similar performance on their own, using them together did increase classification performance from 81% with microelectrodes alone and 84% correct with macroelectrodes alone to 88% when using both in a 1 of 5 task.

Table 1.

Electrodes classified into different brain areas

	Motor (M1)	Sensory (S1)	Parietal)	Mesial
P1	7	7	8	11
P2 (macro)	10	6	13	-
P2 (micro)	17	22	16	-
P3	16	8	16	-

Performance values in Figure 4 denote a small number of individual datasets, so it is difficult to establish confidence intervals. However, primary motor cortex did outperform primary sensory cortex (p<0.01, t-test), and there was substantial variation in the distribution of information across the participants. For participant P2, performance was dominated by parietal areas, posterior to S1. While it is unclear whether these signals are primarily sensory in nature, Figure 3, bottom left, shows high decoder performance from a sensory experiment with this participant, in which the instructor pressed on the passive fingertips (100% correct). P1 had electrodes implanted in mesial areas, potentially near supplementary motor area, which also provided information. Overall, the availability of any information in sensory areas raises the question of how we can best control for this effect, since it will not be available in amputees and spinal cord injury patients.

We evaluated these controls classifying 1 of 3 movements with various combinations of the hand postures to increase the statistical power. The simplest control we attempted was asking the participant to simply imagine movement. Performance does remain above chance using imagined movements, as shown in the red bar in Figure 5 compared to the blue bar which shows normal movements, but increases errors by a factor of 2.7. Another potential control is shown in green, where information is only integrated until movement onset. This also significantly increases the error rate by a factor of 2.4. This allows for the use of motor signals from real movements, but may prevent sensory signals from impacting the decode. Simply eliminating electrodes posterior to central sulcus results in the performance shown in light blue, though this cannot control for sensory information present in motor areas. Applying this control left classification performance largely unchanged in participants P1 and P3, but increased the errors by a factor of 2.1 in participant P2, who had extensive sensory coverage.

Figure 5.

Percentage correct when trying to classify various combinations of 1 of 3 movements in all 3 participants. Color denotes various potential controls for sensory information: no-control (dark blue), integrating log gamma power only until movement onset (green), imagining the movement only (red), and removing electrodes posterior to central sulcus, which includes primary sensory areas (light blue). Chance performance is shown in black. Black error bars denote standard error based on a one way ANOVA. Gray error bars denote standard deviation. Pre-movement and imagination controls have performance significantly below the no-control condition. (p<0.01).

3.3. Robustness

For amputees and C5–C7 spinal cord injury patients, the preserved proximal arm movement makes BMI based grasp control a useful assistive goal. However, this does require that the neural decoder provide accurate information about grasp while the rest of the arm is moving. The first question one might ask in terms of robustness is whether the somatotopy identified with finger movements is roughly conserved upon moving to grasping tasks. Figure 6a on the x-axis shows the normalized difference between thumb/index finger log gamma power and middle/ring finger log gamma power for all participants. If somatotopy is somewhat conserved in grasping movements, one would also expect the normalized power difference between pinching and pointing to covary with the finger power difference, which it does, as shown in Figure 6a (ρ=0.68, p<<0.01, t-test). Example electrodes with a correspondence between finger and grasping movements are shown in Figure 6b. In the left example, higher ring finger log power is associated with higher pointing log power. In the right example, higher index finger log power is associated with higher fist and pinching log power.

Figure 6.

(a) Correlation between finger somatotopy and grasp somatotopy. X-axis corresponds to normalized difference between combined thumb-index finger log gamma power minus middle-ring finger log power. Y-axis corresponds to normalized difference between pinching log gamma power minus pointing log power. Color corresponds to the three participants with P1-blue, P2-green, and P3-magenta (b) Average log gamma power as described in Figure 2 from two example electrodes which show some somatotopic correspondence between tuning to particular fingers as well as tuning to grasp types that involve those fingers. Error bars denote standard deviation.

Next, we examined the specificity of these signals when uncharacterized arm movements occur. One simple observation is that electrodes with increased log gamma power for one finger tend to also show significantly increased power for different fingers as well. For example, if an electrode has significant modulation during thumb movement (p<0.05, t-test, Bonferroni corrected), that electrode has a 93%, 86%, and 82% chance of having significant modulation during index, middle, and ring finger movement. This is consistent with having somatotopic trends in cortex, but no strict mapping. Consequently, if one hand posture is left out of the training data that includes the other grasps and rest, that particular posture is classified as a movement rather than rest 95% of the time. The only exceptions were 50% of the finger splay movements in participant P2.

This underscores the importance of including other extraneous arm movements in the training data. While it would be ideal to examine a range of movements, patients undergoing inpatient monitoring have a limited range of motion and limited time. Therefore a simple wrist flexion and extension task was performed simultaneously with the 8 finger and hand postures, with results shown in Figure 7a. Compared to normal performance when training and decoding from 4 isometric hand postures, if that same decoder is applied while the wrist is moving, the errors increase by a factor of 2.6. This is not simply due to a baseline change, since there was not a statistically significant increase across channels in log gamma power during “rest” trials that included a wrist movement, which did not involve an isometric “squeeze”. The effect of wrist movement can be mitigated by training and testing on data that includes wrist movements, and performance then rises to 84% of its former value. The lack of a complete recovery suggests that wrist movements might somewhat limit the information content about finger movements in the data. Figure 7b shows the log gamma power for various movements with and without simultaneous wrist movement on the same behaviorally-tuned electrodes shown in Figure 2. Power for individual movements may show less separation while simultaneous wrist movements are being performed.

Figure 7.

(a) Percent correct classifying 1 of 4 hand postures and rest (chance 25%) in all 3 participants. To combine data across participants, performance values were normalized to the average performance value from a given participant. Blue corresponds to baseline data during isolated isometric hand postures. Green shows percent correct when a model generating during normal isometric hand postures is used while the wrist is simultaneously exing and extending simultaneously with grasps. Red column shows percent correct when the decoder was trained (and tested) on hand postures that included wrist movements. All data is cross validated on a per trial basis. Asterisks denote a significant decline from baseline in blue (p<0.01) based on a one way ANOVA, and black error bars denote the standard error. Gray bars denote standard deviation. (b) Average log gamma power as described in Figure 2 from the same two behaviorally-tuned electrodes. Black bars denote power during normal isometric movements. Red bars denote power during same movements while the wrist was simultaneously exing or extending.

We next examined robustness through time. In our participants, we were able to record nominally identical isometric finger and hand postures across multiple days. For P3, electrodes were slightly moved during a medical procedure between sessions, so the following numbers include only P1 and P2. Similar to the literature, training a model on 75% of the trials from one day, and testing on either the last 25% on the same day or a different day results in 99% correct classification on the same day and 95% classification on a different day, with no significant difference. However, if we classify all 8 different hand postures, performance drops from 65% correct to 26% correct on a different day. This represents a limited number of datasets (5 same day and 5 different day comparisons) but is a significant decline (p<0.01, t-test).

3.4. Online Prosthetic Hand Control

In one brief session each with participants P2 and P3, neural decoders were generated immediately after isometric tasks were completed in the hospital room, and downloaded onto the real-time system capable of controlling the iLimb Pulse prosthetic hand. Neural data were then interpreted online as the participants completed hand postures, and results were output to the prosthetic hand at a fixed time during the trial (6.5 seconds). The decoder performed well above chance (97% correct classifying 1 of 2 in two subjects, 55% correct classifying 1 of 5 in P2 and 57% correct predicting 1 of 3 movements in P3) but below offline values. Potential reasons for the discrepancy will be discussed below.

4. Discussion

By recording ECoG activity and hand movement data from patients undergoing epilepsy monitoring, we are able to generate datasets that allow us to explore the information content about hand posture in ECoG signals. With these datasets, 4 isometric hand postures and rest were successfully decoded 79% of the time. One prior ECoG study successfully classified 2 different grasp types and rest [36]. The present study supports those results, using a total of 9 movement types, and 4 task variations. In particular, two hand postures versus rest could be decoded offline with 87% correct. Present day myoelectric control systems for prosthetic hands rarely use more than 2 independent control sources [46]. Even with state of the art myoelectric hands, there are a number of auxilary factors that have limited the adoption of this technology by many amputees [6, 7], which may suggest alternative control sources are needed. Also, there is a large population of spinal cord injury patients with arm function, but diminished hand function from whom myoelectric control signals cannot reliably be obtained [4, 5].

The present study is consistent with similar ECoG hand studies to date. Overall, classification rates were similar to other studies [34, 36], and performance differences could be attributed to disparity in coverage areas in the small number of participants evaluated. All classification analyses were also completed using a linear discriminator with regularization, similar to [36, 47], which changed the results by no more than a few percent. This similarity suggests that the data were linearly separable, and that assumptions of Naive Bayes were reasonable. Specifically, the data is well modeled by normal distributions conditioned on movements and the assumption of independence between channels. Also, a linear support vector machine was implemented and compared with Naive Bayes for classifying all possible pairs of 2 movements, and the mean performance between the two was less than a percent. This suggests that classification performance was not overly affected by outlying data points.

We did encounter challenges when attempting to control a prosthetic hand online. These could have been caused by signal non-stationarity or insufficient training data. While we were unable to replicate our offline performance in the online setting, we did successfully decode fist versus rest correctly on 97% of online trials, and a larger number of movements at a rate substantially above chance. Further work is required to determine the difference between the offline and online context, such that it can be mitigated, and high performance can be demonstrated online as well.

This study also suggested that there is some nonstationarity in the signals that is revealed when applying decoders generated on different days. These results are consistent with prior studies suggesting that ECoG signals are stationary over time, for the following reasons. As discussed above, Ojemann and colleagues [48] demonstrated stable classification of 1 of 2 movement types with similar model parameters across days. Results presented here are similar. In general, most datasets showed maximum performance, often 100% differentiating a movement from rest. Classifying 1 of 5 signals relies on subtler differences, and may be more sensitive to small changes in the movement tuning. Another study shows ECoG signals maintaining a correlation between hand position across 5 months. This correlation does appear to vary between 0.4 to 0.95. Also, that study may rely on the burst of gamma power that occurs during movement (and thus will correlate well with movements that always deflect in the same direction), rather than the finer differences in gamma power than seem to differentiate fingers and grasps. Assuming that the observed non-stationarity in the present study does not result from movement of the electrodes, it could be the result of small changes in behavior, posture, fatigue, or medication. There could also be noise differences based on wire placement from day to day, though it is notable that similar non-stationarity was observed in the microelectrodes and macroelectrodes in one participant, when their recording paths were substantially different. Most of these issues could be relevant in a real world clinical setting.

Several potential challenges for hand posture classification in patients without the use of their hands were also explored. In particular, through examining sensory only tasks, and separating the contribution of sensory brain areas it was found that some amount of sensory information was inflating decoder performance. Although some amount of neural activity may be produced in sensory cortex purely by imagining movement, it cannot be assumed that this signal will be present in an amputee or paralyzed individual. Several potential controls were presented reflecting varying levels of conservatism. Imagination by itself had very low performance, but may underestimate the amount of neural activation that for example an amputee could produce by ’attempting’ to move, rather than imagining movement. In Figure 5, the motor-areas only performance may include in some sensory information, whereas the pre-movement only performance may eliminate some motor signals. Therefore, the correct estimate of performance may be between the two. This does suggest that any decode that includes sensory information should be treated as an overestimate of performance. It is however possible that the isometric tasks, though intuitive for prosthetic control, may have more sensory content than tapping tasks in previous literature, due to force related feedback. Overall, the error rate approximately doubled, averaging across the different potential controls. However, there was wide variation between participants, presumably based on coverage area.

Another challenge examined was extraneous arm movements, since many people without the use of their hands retain proximal movement. The general increase in gamma power with movement suggests that previously unseen movements will usually be interpreted as a movement. Also, adding wrist movements reduced the hand posture decoder performance to 59% of its baseline value, and tripled the errors. This occurred even though the brief wrist movement itself was not decodable in the ECoG signal, and therefore likely comes from performing the same grasp in a different posture. This may be an important area for future study. Ideally, systems could be trained and tested using movements closer to activities of daily living, such as moving food from a table to the mouth. More invariant features of data other than raw log gamma power values could be identified. Time series features could be examined with a classifier that could better take into account covariance in the data. It is also possible that extended use of these systems would enable users to practice generating consistent patterns of neural activity for accurate control, which could also address non-stationarity issues.

Several limitations were not explicitly addressed in this study. For example, the classification method used cannot be used for proportional control of joints. Also, long integration windows of 2 seconds were used to minimize difficulties with variable start time, even though classification primarily relied on shorter bursts of activity at the onset of movement. For a BCI system, no time locking signal would be available, and movement times would have to be inferred from the data. In addition, the use of a 2 second integration window precludes the use of this method for a practical BCI system. The gamma-beta power reversal during movement might be a useful data driven marker for movement onset. Even with gamma power alone, in our data, rest could be distinguished from movement with 99% accuracy in general using 200 ms time bins, and 92% accuracy when differentiating 200 ms prior to move onset from 200 ms after move onset. However, discrete commands inferred using this method would not correspond to sustained grasping. A separate command would be required to initiate a grasp, followed by a finger splay command to release it, similar to existing prosthetic systems.

One challenge that is difficult to address in epilepsy patients is demonstration of this level of performance using a less invasive device than a large subdural ECoG grid. A device that is implanted in a manner similar to a skull screw and does not puncture the dura (i.e., decoding epidural signals) might be sufficiently low-risk for a highly functional person like a hand amputee to consider adoption. However, while the electrical impedance of the dura may be low, the increased and varying amount of fluid between the electrode and the brain may cause a larger problem. This could theoretically be mitigated by a hydrophilic electrode substrate, which could remain attached to the dura in a single location [49]. For this technology to become compelling to amputees, the most important challenge is surpassing the level of accuracy or intuitiveness of myoelectric or body powered controllers for prosthetic hands during activities of daily living. This represents a minimum bar for adoption since such systems are both less expensive and safer.

However, there are also opportunities to improve the system that have not yet been fully explored. Targeting of brain areas in motor regions that are highly active with these particular hand movements could be achieved with fMRI. These areas could then be accessed with higher density microelectrodes, which did improve performance in one of our participants. While microelectrodes showed similar information content to macroelectrodes on a per channel basis, combining the two improved decode performance overall, suggesting that an even higher density of electrodes may lead to better results. Also, a wide variety of brain areas can be considered for ECoG electrodes, that have not been traditionally accessible to intracortical studies which focus on motor cortex. Overall, ECoG based devices have the potential to provide a viable source of prosthetic grasp selection signals for a wide range of people without the use of their hands.