Towards Electrocorticographic Control of a Dexterous Upper Limb Prosthesis

Abstract

One of the most exciting and compelling areas of research and development is building brain-machine interfaces (BMIs) for controlling prosthetic limbs. Prosthetic limb technology is advancing rapidly, and the Johns Hopkins University/Applied Physics Lab (JHU/APL) Modular Prosthetic Limb (MPL) permits actuation with 17 degrees of freedom in 26 articulating joints. There are many signals from the brain that can be leveraged, including the spiking rates of neurons in the cortex, electrocorticographic (ECoG) signals from the surface of the cortex, and electroencephalographic (EEG) signals from the scalp. Unlike microelectrodes which record spikes, ECoG does not penetrate the cortex and also has higher spatial specificity, signal-to-noise ratio, and bandwidth than EEG signals. We have implemented an ECoG-based system for controlling the MPL in the Johns Hopkins Hospital Epilepsy Monitoring Unit, where patients are implanted with ECoG electrode grids for clinical seizure mapping and asked to perform various recorded finger or grasp movements. We have shown that low frequency local motor potentials and ECoG power in the high gamma frequency (70–150 Hz) range correlates well with grasping parameters and they stand out as good candidate features for closed-loop control of the MPL.

I. Introduction

An estimated 541,000 Americans were living with some form of upper limb loss in 2005, and that number is projected to more than double with an aging and growing population by 2050 [1]. Loss of limb may occur congenitally or due to cancer, diseases of the vasculature, or trauma—including industrial or farming accidents and battlefield injuries. Recent wars in Iraq and Afghanistan have resulted in a large veteran population with substantial upper limb loss due to trauma. This population has inspired research on the development of advanced prosthetic limbs. One outstanding example has been the JHU/APL Modular Prosthetic Limb (Fig. 1a), developed under the sponsorship of Defense Advanced Research Project Agency (DARPA), has 17 controllable degrees of freedom in 26 articulating joints [2]. This limb has actuators to control the shoulder, elbow and wrist in addition to the fingers and thumb, providing extensive dexterous capabilities. Such an advanced limb also poses a control problem. Traditional approaches have used myoelectric signals from the forelimb of trans-radial amputees. Another more recent approach has been the use of peripheral nerve re-innervation of the chest, using orphaned muscles as a biological amplifier for nerve signals to control a prosthetic limb [3].

Fig. 1.

JHU/APL Modular Prosthetic Limb and ECoG brain-machine interface (BMI) schematic. (a) A photograph of the JHU/APL MPL. (b) The configuration depicted involves acquisition of ECoG signals from electrodes placed on a human brain (left and top), their computational analysis and modeling (right) to drive a prosthetic limb (bottom).

Despite these well-accepted approaches, there is good reason to believe that it is possible to achieve direct neural control of prosthetics that is intuitive and adaptive, involving the subject’s full sensory, motor, and cognitive capabilities. The broad area of research known as brain-machine Interface (BMI) is attempting to leverage patients’ still-functional brains for direct control of a machine, be it a prosthetic hand [4], computer cursor [5], or wheelchair [6]. The goal of BMIs is to interject a machine into the anatomical pathways of the human nervous system to augment, alter, or replace a lost biological function. A basic schematic of a BMI is shown in Fig. 1b.

II. Methods

A. Neural Data Acquisition

The BMI community has explored many different avenues of access to neural signals for BMI applications, but traditionally four modalities dominate: 1) electroencephalography (EEG) is the measure of neural potentials arising from the cortex from electrodes placed on the scalp, 2) electrocorticography (ECoG) is the measure of cortical potentials from the surface of the cortex, 3) local field potentials (LFP) are the low-pass filtered (e.g., 200 Hz) electrical potentials recorded from cortex-penetrating microelectrodes, and 4) single or multi-unit recordings to detect action potentials (or “spikes”) from neighboring neurons. Considering the potential strengths and weaknesses associated with these methods, ECoG occupies unique middle ground among these technological tradeoffs. There have been a few pioneering efforts to use ECoG recording for BMI purposes. These include control of a cursor in one and two dimensions [7, 8] and decoding of individual finger movements [9], slow grasping motions of the hand [10], and grasp type [4]. Two qualitatively different features of the ECoG signal are emerging from these studies. Power in the high gamma band (>70 Hz) has been established as a reliable index of cortical processing [11, 12] and the recently reported local motor potential (LMP) [13] has been used for decoding slow grasping motions of the hand [10] and individual finger movements [9].

B. System Implementation

The system we have developed and continue to refine is designed to enable communication between and synchronization of three distinct nodes. In general terms, these nodes are responsible for neural signal acquisition and processing, behavioral kinematic acquisition, and artificial limb actuation. Neural signal acquisition is accomplished using Neuroscan (Compumedics; Charlotte, NC) SynAmps2 hardware that can be used to amplify either EEG or ECoG signals. For our ECoG experiments, neural signals are sampled at 1000 Hz with a bandpass filter from 0.15 Hz to 200 Hz. Neuroscan SCAN software streams the raw neural data samples over TCP/IP, where they are received by our custom MATLAB (MathWorks, Inc.; Natick, MA) code and processed to extract signal features relevant to human motor movements. Raw neural signals are first re-referenced to a common average reference (CAR) in the time domain as a spatial filter to remove elements of the signal common to all channels. Time and frequency domain features are then extracted from the CAR-filtered channel data. Specifically, the signal power is extracted in five physiologically relevant frequency bands (i.e., μ band, 7–13 Hz; β band, 16–30 Hz; low γ band, 30–50 Hz; high γ band, 70–100 Hz and 100–150 Hz) using the Fast Fourier Transform (FFT) and two amplitude time windows (i.e., 512 ms, 2048 ms) using moving average filters. These features are extracted approximately every 40 ms and synchronized with streaming behavioral kinematic data.

Behavioral kinematic data acquisition is accomplished using the Optotrak system (Northern Digital, Inc.; Ontario, Canada) and the CyberGlove (CyberGlove Systems, San Jose, CA). Artificial limb actuation is achieved either in three-dimensional virtual or physical space. JHU/APL has previously reported and demonstrated the Modular Prosthetic Limb (MPL), a 27 degree of freedom prosthetic arm complete with control of shoulder, elbow, wrist, and fingers. This arm has been duplicated as a virtual model in the Musculoskeletal Modeling Software (MSMS) simulation environment [14], which has been developed at the University of Southern California and is freely available online. The computational resources necessary to process the incoming neural and kinematic data are contained within a single eight core Dell Workstation with 32 GB RAM, of which four are dedicated to MATLAB’s Parallel Computing Toolbox. A photograph of this environment including a patient seated in his hospital room is depicted in Fig. 2b.

Fig. 2.

Photographic example of ECoG array and patient room setup. (a) An intraoperative photo of an ECoG grid being placed in a human patient. (b) A real object target is presented (from bottom right) as a cue to a patient (off-screen to the left for anonymity). The patient is pointing to the target, and his motions are being tracked by Optotrak markers on the shoulder and hand. The three-dimensional position of the patient’s hand and cue are being displayed in the MSMS simulation environment. The virtual cue is yellow, indicating a successful trial.

III. Results

We have used the system described to initiate research into ECoG-based control of a dexterous prosthetic limb. ECoG electrode grids are predominantly implanted for clinical purposes in patients with uncontrollable epileptic seizures (Fig. 2a). In previously published work [10], our lab discovered that the LMP recorded from subjects implanted with ECoG grids could be used to decode slow grasping motions of the hand with simple linear models. LMP signals with highest correlation to the recorded kinematics were selected for inclusion in the decoding models. Peak decoding performance was achieved with as few as four electrodes in areas that can be identified intraoperatively as having motor involvement, meaning that these signals can be recorded from low-footprint ECoG grids implanted in known areas. These results are very promising for the use of LMP signals for neuroprosthetic applications. The robustness of LMP as a phenomenon is validated by the high decoding accuracy across sessions.

In more recent work from our lab, we investigated the neural signals responsible for the coordination of slightly more complex grasps [15]. Our study showed that frequency components in the high gamma band (70–100 Hz and 100–150 Hz) provide the best performance for decoding grasp aperture. Fig. 3a shows the location of the implanted grid electrodes, with darkened electrodes corresponding to motor brain areas as identified by ESM. Fig. 3b shows the spatial pattern of decoding accuracies obtained using 70–100 Hz power from single electrodes at various locations on the cortex. Again, the highest-performing electrodes appear to be concentrated over areas identified as having motor involvement prior to experimentation. Fig. 3c demonstrates correspondence between observed and decoded grasp aperture traces using the twenty features that best predict grasp aperture in each cross-validation training set.

Fig. 3.

Spatial distribution of single feature decoding accuracy and example decoding trace with twenty features. (a) Circles denote implanted electrodes that were included in the analysis, while darkened electrodes indicate that motor behavior was elicited or interrupted during electrocortical stimulation mapping (ESM). (b) Single feature decoding accuracies: Pearson’s correlation r between observed and decoded traces. (c) The example traces show the fidelity of decoded grasp aperture to observed grasp aperture. Predicted traces have been formed in fivefold cross-validation with linear models trained with twenty distinct neural signal feature inputs. (Adapted and modified from [15].)

Our results not only indicate that complex movements can be decoded from a patient’s ECoG signal, but that both LMP (an amplitude feature) and high gamma band (a spectral feature) should be considered in decoding complex motor tasks. While it is an area of active investigation, it is our hypothesis that the LMP, as a slower signal, encodes information about low velocity or repetitive movements fairly robustly, while the high gamma band may be more useful for decoding movements with higher degrees of complexity or more sudden onset.

IV. Future Directions

We are making steady progress toward the dream of neural control of prosthetic limbs using a variety of means, but the journey is just beginning. A few major challenges to achieving ECoG based control of dexterous prosthetic remain, including: 1) improving the resolution of ECoG arrays; high resolution ECoG with arrays of mini- and micro-electrodes may provide better localization to the areas of the cortex responsible for dexterous hand and finger movements; 2) maturation of decoding algorithms specifically suited to ECoG signals; ECoG signals in very low frequency as well as high gamma bands may offer novel decoding capabilities and information; 3) provision of proprioceptive and touch feedback to the neuroprosthetic user by stimulating intact peripheral nerves or directly stimulating somatosensory cortex [16] may greatly facilitate natural control of an artificial limb; 4) building fully implanted ECoG systems—long-term cortically-controlled prosthetics will need to be comprised of electrodes, circuits, and telemetry interface to the limb while being fully implanted and powered; 5) ethical considerations in selection of patients and implantation with regards to the potential risks of benefits to each individual patient.