Implants and Decoding for Intracortical Brain Computer Interfaces

Abstract

Intracortical brain computer interfaces (iBCIs) are being developed to enable a person to drive an output device, such as a computer cursor, directly from their neural activity. One goal of the technology is to help people with severe paralysis or limb loss. Key elements of an iBCI are the implanted sensor that records the neural signals and the software which decodes the user’s intended movement from those signals. Here, we focus on recent advances in these two areas, with special attention being placed on contributions that are or may soon be adopted by the iBCI research community. We discuss how these innovations increase the technology’s capability, accuracy, and longevity, all important steps that are expanding the range of possible future clinical applications.

Introduction

Voluntary limb movement in primates, including able-bodied people, is achieved in part by the conveyance of high level motor commands from neurons in the motor cortex to neurons within the spinal cord. In response to input from these so-called “upper motor neurons”, as well as inputs from other descending systems and cells local to the spinal cord, alpha (“lower”) motor neurons then activate muscle fibers generating motion. Injury or disease, such as brainstem stroke or loss of a limb, can disconnect the cortex from its effector target. Ongoing research is focused on building technological bridges between the motor cortex and external devices in order to restore motor function. When these systems work as designed, the intent to move again becomes translated into overt action.

Variously labeled as neural interfaces, neural prostheses, brain machine interfaces (BMI) or brain computer interfaces (BCI), this technology promises to be useful to a large number of people. In the United States alone, it is estimated that 1.6 million Americans live with limb loss (1) and 5.5 million people experience some form of paralysis (2) including 50,000 individuals who have complete tetraplegia (severe functional impairment or paralysis of both arms and legs) (3). Commonly, people with tetraplegia rely on others to perform routine, everyday actions. Responding to the need for better assistive devices, BCI technology could provide a control signal for any range of devices: a computer, robotic or prosthetic arm, or a powered wheelchair. For a true restorative function, people suffering from paralysis of an arm might, in the future, be able to recover control by re-animating the affected muscles via brain-controlled functional electrical stimulation (FES); others with limb loss could employ a robotic prosthesis that emulates arm and hand actions.

This review focuses on intracortical brain computer interfaces (iBCIs) where, for example, microscale probes having 96 fine tipped microlelectrodes in a compact 4×4 mm array are inserted into the cortex (reviews of scalp-based electroencephalography and brain surface-based electrocorticography BCIs can be found in (4) and (5) respectively). The sensors are sensitive enough to pick up the discrete all-or-none output of single neurons, the action potential, commonly referred to as a “spike”, as well as the summed voltage fluctuations from small to large numbers of neurons, called field potentials. Each electrode provides spiking from up to a few neurons (typically 1 or 2), yielding the population's time evolving output pattern. These represent but a small sample of the entire set of neurons in this limited region, as spiking can only be detected by microelectrodes closely approximated to a neuron.

Spikes are desired because they are the information-rich signal nearly universally employed by neurons for communication in the brain. Beginning approximately 50 years ago, experiments by Evarts and others helped to reveal how individual cortical neurons, recorded one at a time, control voluntary movement, helping to lay the groundwork for current iBCI work, e.g., (6–11). Multielectrode approaches helped to provide (12, 13) early examples of real-time prediction of an animal’s wrist position from the decoded motor cortex output (14, 15), and enabled animals to perform simple tasks via an external device (16). The first clinical realization of these technologies occurred in 1998, when a person with amyotrophic lateral sclerosis (ALS) paralysis controlled an on-off switch using two electrodes implanted in the motor cortex (17).

Based upon the many years of publicly funded basic science, research beginning in 2000 demonstrated that monkeys could use motor cortex spiking patterns to control a computer cursor in two or three dimensions (18–22). Soon afterwards, in 2004, the first demonstrations of intracortically-directed 2D cursor movements and simple robotic control were accomplished by people with tetraplegia using an iBCI (23). This was followed by iBCI-enabled 3-D control of a prosthetic arm by non-human primates (24). Most recently, multi-dimensional robotic arm control was successfully achieved for reaching and grasping actions by two people with tetraplegia (25). The future promise of brain-controlled FES devices was further demonstrated by iBCI-driven muscle control in monkeys with temporary arm paralysis (26–28), further underscoring the potential highlighted in an initial demonstration by a human with paralysis using an iBCI to control a simulated, dynamic FES system and arm (29).

Modern iBCIs have several major components, illustrated in Figure 1. The user imagines or attempts a movement, resulting in motor cortical neuronal activity. An implanted multiprobe sensor of electrodes, commonly referred to as a microelectrode array (MEA), records neural activity in the form of extracellular voltage signals. Data acquisition (DAQ) hardware transports, amplifies, and digitally samples these signals. The data is processed on a dedicated, real time hardware and software platform, which hosts various decoding routines. In the typical real-time data processing stream, neural signals first are processed to extract informative features related to intended movement, after which estimation techniques translate those features into device commands. The device then carries out the desired action as well as provides feedback to the user.

Figure 1.

Illustration of an iBCI setup.

The core elements of an iBCI (Figure 2), from neural signal collection to actuation of the effector, each require the iBCI neuroengineering team to make important decisions. Each of these elements (which could also be broken into several smaller elements) also presents opportunities for focused research and development. We take the engineering perspective in this review, where the domain of decoding encompasses the combination of the feature extraction, estimation, and calibration elements (30); this viewpoint is used to organize subsequent sections.

Figure 2.

Block diagram of iBCI architecture. Arrows indicate the major flows of information whereas the red-shaded box contains elements associated with decoding.

Prior to use, the BCI must be calibrated. Plainly stated, a map must be built between the observed neural activity associated with intended movement and the actual movement of the effector. First, the user (with, for example, tetraplegia due to stroke or ALS) engages in a series of instructed behavioral tasks where various real world actions, such as the preprogrammed motion of a computer cursor, are observed while imagined or attempted by the user (who is unable to make them). Next, with both the neural signals and the attempted (or, at least, instructed) movements being known quantities, calibration techniques tune the decoder. It is important to note that during this process only the decoder has been calibrated; no real training or learning has (yet) occurred from the user’s standpoint. While reasonable to believe that users presented with a new and reasonably useful tool (an iBCI) will become more proficient in its use over time (31), important questions regarding recorded signal stability as well as the potential for simultaneous co-adaptation (32) of both the decoder and the user yield the clear demonstration of such learning to future research.

A thorough review of the several active research areas highlighted in Figure 2 would extend far beyond this review (see (33) for several excellent chapters in this regard). Here we focus on recent advances using the implanted sensor as well as the state-of-the-art in decoding algorithms. In each case, the emphasis is on results that have been or we believe are likely to be widely adopted by the iBCI research community.

Implant

Microelectrode arrays

Overview

From an electrical and electrophysiological point of view, microelectrode arrays (MEAs) for iBCIs need to be designed considering details of both the electrode-brain tissue interface and the macroscale (i.e. lumped) circuit equivalents. Quantitative modeling for optimizing capture of the neuronal electrical signals is complex, influenced by the role of the electrolytic bilayer at the interface between the electrode recording surface and brain tissue, electrode tip shapes, potential tissue reactions, etc. (34). Metals such as Pt , W, and Ir are chemically stable in the electrolyte-rich and proteinaceous environment of brain parenchyma and have reasonable work functions to match the Na+, K+, Cl−, Ca++, and Mg++ concentrations that dominate. To capture both spikes and lower frequency field potentials, the equivalent circuit of the single microelectrode/tissue recording interface should typically cover a bandwidth of ≈ 0.5 Hz – 10 KHz. A single action potential might generate recorded amplitudes from few tens up to >100 µV, registered by an electrode of ~ 500 kΩ in impedance (i.e. sub-nA currents), thereby framing the design requirements for subsequent analog preamplifiers (gain, noise figure etc.), digitizing circuits, and the downstream signal processing and data management electronics. Note that apart from fundamental sources of noise common to all electrical systems (Johnson noise, etc.), the brain operates within its own inherently “noisy” background (since its circuits are continuously ‘alive’), thereby adding to the challenges of MEAs acquiring consistent neural signals that correlate with specific cognitive process or behavior. Another set of important contemporary questions is how reaction by the body tissue on implantable electrodes and related device structures affects their truly long term chronic performance (≫ 1 year).

Early intracortical microelectrode arrays composed of bundles of wires have been now largely replaced by ‘monolithic’ arrays for work in primates. Of these, we mention the Si-based arrays where lithographic and electrochemical techniques are combined to fabricate tapered microscale ‘beds of needles’ (35–37). For example, in the so-called “Utah” MEA (inset of Figure 3), each approximately conically-shaped 1–2 mm long electrode, with its p-Si shank insulated by biocompatible parylene, is coated at its tip by Pt or Pt/Ir. The heavily doped p-Si provides a low-loss conducting path onto the silicone insulated, planar support substrate wherefrom a wirebonded bundle (say, 100 insulated 1 mil Au wires) transports the recorded neural signals through the skull and the skin to exterior electronics (Figure 3). Operationally, specialized neurosurgical techniques are employed to access cortical areas of interest, with MEAs inserted into the brain typically using calibrated pneumatic single-impact pulsed drives. While these MEA implants have shown their mettle as chronically viable implants as described in this review, further improvements in their reliability and further control of their critical dimensions on the micrometer scale precision are expected both from improvements to current manufacturing approaches as well as innovations in both materials and advanced microfabrication (such as wafer scale processing (38)).

Figure 3.

A silicon-based cortical MEA (inset); implanted for intracortical neural recording via a percutaneous connection to a skull mounted pedestal connector.

Based on silicon substrate shanks, the “Michigan array” can run multiple connective lines down to pads of iridium along a shank to achieve multi-depth electrodes (37). One shank can efficiently support as many as 16 electrode sites at different depths, and can also horizontally cluster electrodes in placing several at the same depth along a single shank (39). Conducting polymer nanotubes have also been used to dramatically decrease impedance when attached to the surface of the electrode sites (40), potentially enabling the creation of thinner but still functional recording electrodes (41) which can be more densely placed in cortex without overly disturbing the surrounding tissue. Yet another technology, the “neurotrophic” electrode (17) aims to provide for long-term intracortical recordings by attracting the growth of neurites into a wire electrode-embedded glass enclosure.

Wireless implants

Several successful clinical technologies have benefitted from having percutaneous components during their initial development and testing (e.g., cardiac pacemakers, deep brain stimulators). Nevertheless, there are a number of aesthetic and safety benefits to developing systems which dispense with an externalized “tether”, and which ultimately become fully implanted. One current research trend focuses on efforts to integrate the intracortical microelectrode recording arrays with wireless electronics so that the physical cabled connection between a subject and external electronics and external assistive devices can be removed. Wireless approaches will greatly improve the mobility for a subject, be that in human neuroprosthetics applications or in freely moving non-human primates in fundamental neuroscience research. Many groups worldwide are now working towards wireless active microelectronic neural interfaces, especially focusing on very low power, application-specific system-on-chip integrated circuits (ASIC/SoC) which consolidate the analog, digital, and telemetric components to a wearable system, respectively. Several sophisticated designs have been demonstrated at the benchtop level. Yet, much of this integrated ASIC-based engineering work still awaits transition to in-vivo use for primates. One successful strategy has been to retain the percutaneous wiring from the MEA to the skull-mounted connector pedestal and affix battery-powered head-mounted external modules on these pedestals in non-human primates, to demonstrate transmission of multichannel neuronal data as a broadband radiofrequency digital stream (42–45).

Transcutaneous systems

The emergent wireless approaches to iBCIs have the goal of eventual implementation as fully implanted electronic microsystems whereby neural signals from MEAs are amplified and transmitted from hermetically sealed modules which are embedded entirely within the body (head, or head and chest), without any skin-penetrating wiring. The ability to employ fully wireless, and eventually bidirectional, iBCI interfaces that are enveloped by the body’s most potent protection against infections etc., namely the skin, opens up potentially entirely new vistas for next generations of neurotechnologies. As a state-of-the-art example (ca. 2012), we show in Figure 4 a 100-channel wireless neural broadband implant which has been successfully tested in mobile pig and non-human primate models (D. Borton, M. Yin, J. Aceros, W.R. Patterson, and A.V. Nurmikko, in progress) (46). The device shares a commonality with cochlear and cardiac implants in that the active microelectronics, battery power, and telemetry circuits are encased within a hermetically sealed titanium enclosure. By contrast to these low data rate neural stimulation devices, however, the amount of information which is being acquired wirelessly from the brain and transmitted through the skin to nearby receivers for decoding is huge (on the order of 50 MBits/sec at this writing, and expected to increase in the future). Such demands on data rates not only impact the required sophistication in designing and fabricating the wireless implants, but will require new strategies for handling and storing user-specific neural information in the future.

Figure 4.

A wireless neural interface, hermetically sealed, for untethered neuroscience and potential clinical use. (a) An image of the device after hermetic closure and ready for implant (device placed in special holder for sterilization and transport to the operating room). The standard 10 × 10 element microelectrode array (Blackrock Microsystems, UT) can be seen exiting the enclosure in a 100-wire bundle. (b) In a view through the single crystal sapphire window, used for electromagnetic transparency, the receiving power coil (red) and the wideband data telemetry chip antenna (blue) can be seen. The sapphire is braised to the titanium, maintaining the hermetic seal. (c) The intracortical neural sensor, manufactured at Blackrock Microsystems, and a reference wire can be seen in relation to the supracranial/subcutaneous enclosure. (d) All 100 wires individually enter the enclosure through gold and ceramic feedthroughs, after fanout onto a flexible polymer (Kapton) interconnect board sealed to the (outside) bottom of the titanium enclosure. Here, we show the 100 wire bonding site (grid, right) and the interconnect circuit board soldered to the feedthrough assembly. The interconnect board is packaged in biocompatible silicone rubber to maintain robust electrical isolation of the input electrodes.

Decoding

Neural signal feature extraction

Overview

Once the recorded signals have gone through data acquistion steps, the decoder first performs the step of neural signal feature extraction (Figure 5). During each operational cycle of the decoder (aka time step), a vector of values is computed from the multivariate time series sampled from the many voltage waveforms streaming in from the sensor. One classical approach depends upon the visual inspection of the action potentials (spikes) emanating from individual neurons, e.g. (23), which are a complex time derivative of the extracellular voltage gradients shaped by filter settings. The typically biphasic, ms-long voltage waveforms of each channel are visually inspected to manually set time and amplitude windows, converting them to discrete events in time, permitting detection of spikes and their assignment to a single neuron or a cluster of them. A unit’s spikes are then time binned to compute the spike counts of each unit in each of the time bins, creating a point process time series for each channel.

Figure 5.

Illustration of a decoder's architecture and the flow of information passing through the various components.

Threshold crossings

Sometimes, the shape and size of a unit's waveform can change significantly over time. A systematic analysis of instabilities found several cases where the combination of time-amplitude bins traditionally used in spike sorting failed to detect spikes due to drifts in their amplitude which result from even small movements (on the order of a few tens of microns of the electrode recording site with respect to a particular neuron) (J.A. Perge, M.L. Homer, W.Q. Malik, G. Friehs, J.P. Donoghue, and L.R. Hochberg, submitted) (47). A simpler and more robust approach may be to count the number of voltage threshold crossings within a given channel and time bin (48). Termed threshold crossings, the technique enabled no appreciable drop in decoding performance despite significant reductions in spike amplitudes (47). Use of threshold crossings is gaining adoption with their use a key element in recent pilot clinical studies (25), but success may be sensitive to signal to noise ratio, electrode properties, the size and shape of neurons and other unknown factors.

Multiunit activity

An alternative or addition to detecting spikes analyzes signal fluctuations available in the high frequency band, e.g., between 300 – 6,000 Hz. The “multiunit activity” (MUA) can also be used for iBCI control after passing through signal processing such as a nonlinear filter (49). More recently, in offline decoding, non-linear filtering of MUA was reported to yield better results than either spike detection and sorting or threshold crossing (50). 3D reach and grasp in monkeys was decoded offline by extracting power information in the 200–400 Hz frequency bands (which extends into the MUA range) (51). In preliminary closed-loop, head-to-head comparisons by a person with tetraplegia operating an iBCI, MUA performance was found comparable to spike counts (52). Perhaps most attractive is the evidence given of MUA’s non-stationary statistics being less pronounced than other high frequency signal options (50). Again, caution is needed because these advantages may be modified by amplifier or electrode properties.

Local field potentials

Other frequency bands within the range of signals generated by neural activity also contain information about movement intent. Unlike spikes, the primary sources for a given local field potential (LFP) signal measured in the motor cortex remains unclear. Although it has been postulated that LFPs are the result of synaptic currents, many factors such as spike afterhyperpolarizations can play a role (53). Prominent oscillations, one example of which is the commonly referred to “beta rhythm”, have been linked to attention (54) and found to noticeably decrease in amplitude between movement preparation and onset (11). In the last decade, reach direction was decoded from LFPs in monkeys in offline studies (55–57). The features extracted varied, for example, a Gaussian kernel smoother (55) or a multitaper spectral analysis (57). The frequency bands studied may contain different types of information about intended movement; for example, the average amplitude within the 63–200 Hz band increased at movement onset relative to baseline, whereas it was observed to decrease between 16–42 Hz (56). Three dimensional reach and grasp in monkeys can be decoded, offline, from intracortically-recorded low, midrange and higher frequency bands, with the quality approaching, but not surpassing, that of spikes (58). Figure 6 provides an example from a person with tetraplegia with a sensor placed in the arm area of motor cortex. In the presented spectrogram, there is a decrease in amplitude during intended arm movement in the 12–40 Hz range, with both increases and decreases in power seen at other frequency bands throughout the intended movement.

Figure 6.

Representative spectrogram of the LFP signal in one channel, averaged over 70 repeated trials. Time 0 denotes the instructed onset of movement (dashed line). Colors indicate amplitude, normalized by dividing by average baseline activity prior to movement onset (blue greatest reduction in activity, red greatest increase in activity). The data was recorded from a person with tetraplegia intending to move her own hand during repeated cursor control calibration trials. Note the appearance of three differing bands (approximately 0–3, 3–40, and 40–100 Hz). (courtesy of J. A. Perge, unpublished)

Using two arrays and frequency bands, 0.3–4 Hz and 48–200 Hz, classification of eight target directions was achieved to an accuracy comparable with spikes (59). Other reports decoded natural arm reaches to predict arm endpoint motion and grip aperture (58, 60). Despite these advances, comparisons between decoding with LFPs vs. spikes do not point to LFPs as a clear winner (55, 58). The real value then of the signal is thought to lie in providing more robust features, particularly over the course of time. For example, the ability to detect a spike might be susceptible to array micromotions whereas the aggregate nature of LFPs may mitigate signal changes due to tiny array movements because of their volume conduction properties. This advantage might explain one study’s report of channels that had lost their spikes still retaining most of their predictive power in the LFP features (61).

An alternative use for LFPs lies in identifying discrete states linked to movement. In electrocortical studies, a person’s visual attention (versus eye movement) can be classified above chance levels into one of eight directions (62). In the posterior parietal cortex (PPC), 0–10 and 20–40 Hz bands have revealed the onset of motion as compared to the preparation phase (63). The accomplishment is an especially significant step because the decoding was done online where the monkey used the iBCI to perform the task. Thus, LFPs could play a part in a hierarchal decoding strategy where identification of the discrete motor state dictates how to approach estimating the continuous variables.

Also, the precise features extracted from these broadband neural signals can make a substantive difference in their decoding utility. Instead of LFP amplitude, recent work is starting to reveal the role of LFP phase. Investigations into beta oscillations, for example, have found a statistical dependence between the oscillation's phase and target reach direction (64). These observations are supported by the revealed coherence between beta oscillations and EMG recordings (65). Spiking activity has also been linked to the phase of beta oscillation, where directional information of the former was a function of the latter (66). These new scientific insights suggest that extracting phase information from LFPs might improve iBCI decoding by considering the influence of LFP waveform phase on modulations in spiking activity.

Decoder calibration

Overview

In order to customize the decoder to the user, initial filter calibration exercises engage the iBCI user. In animals, the subject is trained to perform the behavioral task, using overt arm actions. Meanwhile, recordings of the brain activity are collected (16, 18–22). In contrast, people with paralysis are asked to attempt arm and hand motions that would, for example, generate planar ‘mouse’ trajectories and clicks matching the computer controlled cursor on the screen (23, 67). In these calibration sessions, the recorded neural signals and any information processed from them are not presented to the user and thus are called open-loop. Similar approaches can be used for initial training for the control of multi-dimensional reach and grasp movements of a robotic or prosthetic arm (25).

The recorded neural signals and the corresponding movement information are then fed into calibration routines which set parameters within the decoder. For example, when a person with paralysis used an iBCI to control a computer cursor's motion, linear regression processed the calibration data to arrive at the needed matrices for the linear filter (23). In the case of a Bayesian classifier, once the neural signal samples are separated into their respective classes, class conditioned distributions can be fit from the data, e.g., calculating means for independent Poisson distributions (22). The calibration procedures usually extend to processing the features themselves. For instance, from the set of spiking units collected, only those well modulating with reach direction as shown through ANOVA were selected for inclusion into the decoder (21). Such steps help to best fit the decoder's parameters given the calibration data while protecting against over fitting.

Closed-loop calibration

The concept behind closed-loop calibration is to adjust the decoder's calibration after a closed-loop block of trials, one where the iBCI is used to carry out the task. The process has an iterative nature where the previous calibration block adjusts the decoder used in the next calibration block (19). Supporting the approach, differences were found in neuronal activity between actual and observed movement (68, 69). Furthermore, while observation-tuned neurons were found to be predictive of movement direction, strictly action modulating neurons did a better job. Even more recently, such tuning differences, depending upon cognitive context, have also been reported in preliminary studies in clinical trial participants with tetraplegia (70).

Closed-loop decoder calibration has been used successfully to enable control in able-bodied monkeys (24, 71) and in people with tetraplegia (see Figure 7) (25). An innovation is to run an automated software routine to assist epochs of “brain control” (that is, iBCI-controlled movements, as opposed to hand-controlled movements), thereby increasing trial success. In early iterations, considerable assistance is provided, but with each iteration it lessens until the task is completely governed by brain-control. Not only is the technique thought to improve the decoder, but the user’s neural activity appears to change to accommodate the iBCI. In several studies, the response characteristics of units changed as the monkey learned a task (72). When a tuned decoder was intentionally altered while a monkey was operating an iBCI, units’ firing rates changed during continued use of the iBCI (32). In another closed-loop iBCI experiment, rats improved their task success rate without changing their overt behavior (73). Thus, use of closed-loop calibration appears to allow both the user and the decoder to update their internal calculations in an effort to converge on stable configurations for useful closed-loop iBCI operation.

Figure 7.

Illustration of closed-loop calibration technique. (a) White shows cursor trajectory; the blue and red dots show the starting point and goal respectively. At each sample time, the direction between the cursor and goal is calculated (red arrows). This direction is then compared to the current cursor heading in order to generate an error signal with which to adjust the decoding tuning parameters. (b) A similar technique can be applied in the 3D case (where the white coloring is switched to black for contrast). ( (a) courtesy of B. Jarosiewicz, unpublished) ( (b) courtesy of (25), adapted)

Sometimes, long after a stable mapping has been reached, it changes, making the initial decoding model inaccurate. Fortunately, there are mathematical frameworks to formalize possible paths of deviation and ways to fix this problem. On the efficiency side, a recursive least squares formulation can prevent one from having to reconsider all prior calibration blocks (74). Similarly, the problem can be cast in a Bayesian context (75). These innovations get closer to how best to utilize and efficiently process the closed-loop calibration data to adapt the decoder.

Dimensionality reduction & regularization

Several methods seek to remove or discount one or more neural activity features collected during filter calibration in order to make the decoding more accurate. One direct way to go about this is to focus on well-isolated single unit activity (SUA). By comparing changes in waveform characteristics, spike intervals, and directional tuning, one study chose a small subset of 10–15 units from the 75–100 recorded (31). The calibrated decoder performed well over many days (9 days in one monkey, 19 in another).

Others have employed more automated means, such as cross validation, whereby the calibration data is divided into three sets: training, validation, and assessment. First, the decoder training data is used to build models, i.e. the mathematical mapping between features and motor states. These models are then checked with the validation data set to see which performs best and finally the true test of performance occurs with the assessment data. If models under comparison differ by the features chosen, then a greedy method, akin to forward stepwise regression, can find promising, though typically not optimal, subsets (76). This strategy yielded reductions in the number of features chosen, in some cases by a factor of four, in 3D reach and grasp in monkeys (51, 77).

Alternatively, the decoder can treat the neural signal features as being driven by a small number of latent variables, which then represent motor intent – a technique called dimensionality reduction. Gaussian Process Factor Analysis (GPFA), for example, both smooths in time as well as finds a smaller set of latent features (78). Using data from neural ensembles, GPFA was found to improve trajectory reconstruction accuracy over standard methods. Use of dimensionality reduction in real-time decoding reduced the number of variables driving the decoder by approximately 90% in one case and 60% in another (79). Furthermore, error rates in directional classification dropped dramatically from roughly 20 to 5%. Interestingly, such latent features may generate new theories about fundamental, computational principles of the motor cortex. In a reduced space, characteristic, repeatable oscillations emerged, suggesting that cell populations in M1 might not encode movement variables per se, but rather serve as a pattern generator to drive downstream computations and eventually the movement itself (80).

These methods become particularly important if more advanced feature extraction methods are used. With LFP histories, the number of features can grow easily into the thousands. Given such a large data set, many of the features display a redundancy which can be taken advantage of to reject a variety of noise sources such as those of a biological nature and recording artifacts. Dimensionality reduction and regularization is one way to be robust to such disturbances and indeed has been successfully wielded to handle feature deluges of as much as 6,400 features in order to achieve stable performance in ECoG work over many days (81).

Motor state estimation

Overview

Given a set of features extracted from the neural signals, computations estimate the intended motor state. Decoded motor states have focused largely on those surrounding arm reaching and grasping, particularly as these functions are listed as most important to restore by people with high cervical spinal cord injury (82). Specifically, many earlier studies investigated decoding endpoint velocity and/or position in order to control the movement of a computer cursor on a screen (18–20, 23). For discrete selection, monkeys have chosen a reach target position/direction from a finite list of options (21, 22). One end goal of the work is to realize the possibility of enabling someone with paralysis or loss of a limb to control a robotic aid or FES-driven limb for reaching and grasping. The relatively high level motor commands of selecting a target or controlling the endpoint trajectory could then be converted to control commands needed to drive the effector.

A standard estimation approach is the linear filter (83) and it has been used for continuous decoding, e.g., (9, 18, 19, 23, 24). Each feature value specifies the length of a vector of fixed orientation in the motor state space. The vectors are then summed to yield the state estimate. For discrete states, Bayesian classifiers have also proven useful (21, 22). They typically model the feature distributions conditioned on the discrete states as either Poisson or Gaussian. Both methods have the advantage of being calibrated by exact methods, fast online computation times, and relative transparency in the behavior of the tuned estimator.

More accurate estimation models

In a head-to-head comparison, however, between linear and Kalman filtering, the latter was shown better using several standard performance metrics of cursor control (84). Interestingly, as Figure 8 shows, the Kalman filter did not always yield better cursor trajectories. The algorithm performs the estimation based upon linear, multivariate Gaussian models linking the features to the motor state as well as describing the motor state dynamics. The decoder can run on the same time scale as a linear filter since the results from the linear algebra calculations quickly reach steady state (85). Of especial note is the fact that the evaluation was done by people with tetraplegia operating an iBCI to control the motion of a computer cursor. Since then, the method has provided a reliable estimation approach, even in trials 1000 days after implantation of the sensor (67). A variant of the technique recently enabled two people with severe paralysis to perform 3D reach and grasp tasks by way of a robotic arm aid (25), in one case >5 years after sensor implantation.

Figure 8.

A comparison of the paths under brain-control using the Kalman filter vs. linear filter. The paths were performed by a person with tetraplegia using velocity-based decoders during the eight-target center-out task. Performance results from two recording sessions are shown where both the linear filter and the Kalman filter were sequentially tested. Eight targets were located at (0, 45, 90, 135, 180, 225, 270 and 315 degrees). Units indicate the number of features employed by the respective decoding filter. (courtesy of (84))

Careful evaluation of the true functional mapping showed a large fraction (35%) of the unit spike counts to be nonlinearly related to intended motor control states (86) and for this reason, some earlier attempts with iBCIs worked with nonlinear decoders (14). Another line of research added an auto-regressive element to the feature histories (87, 88). By combining the two types of model refinements, animals were able to operate iBCIs within an unscented Kalman filter framework, yielding improved performance (89). Very general mappings between the features and states can be handled by particle filters (48, 90–92) or neural networks (14, 93).

Discrete motor states can also be introduced into the model. The resulting hybrid state space model switches between a series of discrete modes (94). Computational machinery infers the state transitions during the model fitting process via methods such as expectation maximization (EM). The technique was applied to augment the Kalman filter (95) as well as a point process filter (96). The augmented set of hidden motor states, like the nonlinear models, free the estimator from purely linear assumptions should the calibration data show a clear nonlinear relationship. In terms of metrics such as mean squared error, use of hybrid state models have achieved reductions as high as 22% (96). However, the results should be treated with caution as the real test lies in closed-loop, head-to-head comparisons (97).

Improving the noise modeling, another line of inquiry makes the more accurate assumption that the spike counts follow a Poisson process (rather than a Gaussian). This is the territory of point process models and they account for the exact timing of the spiking sequences (94, 98). Unlike most linear filter methods which process spike count features within, for example, 30 millisecond bins (24), point processes desire to isolate each unit firing event in time for a typical time bin of 1 millisecond (99). In offline studies, decoders running point process models have been demonstrated in animals (96, 100, 101) as well as in people (99). The cost for the improved modeling fidelity, however, is that decoding becomes more computationally intensive.

Choice of motor control states

Increasing the degrees of freedom decoded from motor cortical activity would enable more tasks to be performed with an iBCI and/or with a greater degree of control. Reports of 3D reach and grasp with a robotic arm in able-bodied monkeys (24) and in people with tetraplegia (25) provide direct evidence of this and relied on expanding the state dimension from two to four (one extra spatial dimension and a variable for grasp). For more precise control, decoders could first determine whether or not the person is attempting a move and in what general direction. Initial creation of such a first stage classifier was reported in monkeys (102). Even finer stages of motor preparation and execution can be detected with iBCIs, such as shown in macaques in which pre-trial, pre-cue, memory (remembering target location during a delay), and post-saccade phases could be readily decoded (103).

Alternatively, the measured neural activity may better associate with a different set of motor states. For example, a unit could be more tuned to wrist extension rather than general upward motion (104). Responding to the need, researchers decoded 25 joint angles under natural reach and grasp conditions in monkeys (77) under open-loop conditions. The strategy relied on statistical dependencies between the joint angles, allowing the kinematics states to be compressed down to 10. As the 10 variables were estimated at each time step, a linear transform expanded the estimate to a vector representing all 25 joint angles. Thus, even with a relatively small sample of neurons in motor cortex, iBCIs based on motor cortical recordings should enable complex reach and grasp movement. Forces are also encoded in the stream of neural activity measurements (8, 105). In planar manipulandum experiments, software simultaneously decoded force and velocity in offline analyses (106). Estimating intended force (grip force, for example) in people with paralysis will be an important venture, and is likely to benefit from feedback from the effector.

Decoding muscle contractions had led to temporarily paralyzed monkeys regaining control of their muscles through functional electric stimulation (FES) driven by an iBCI. In 2008, monkeys were reported to control wrist torques in closed-loop experiments (26). More recently, five electrodes stimulated three muscles that drove wrist flexion and grip (28). Effectively building an artificial bridge from the central nervous system to the muscles, the animals were able to pick up balls and place them into a receptacle. For people, control of a virtual arm was shown possible by a person with tetraplegia, using a simulation that incorporates the complex, dynamic response behavior of an FES system (29). These advancements mark key milestones towards restoring limb control to those with paralysis.

While the majority of studies in non-human primates have focused on reaching and grasping, research has also been proceeding in other types of motor control. Going beyond hand grip, individual finger movements (i.e., which finger moved) has been correctly classified better than 80% of the time (107, 108) in offline studies. Leg kinematic offline decoding studies in monkeys have also been performed (109), although full closed-loop locomotion presents interlimb coordination and balance challenges. Rats, with electrodes in the hind limb/trunk region of M1, successfully lightened a load applied to their hips, in time with their gait (110). While neither study demonstrated closed-loop control over the entire combination of movements needed to enable walking, they do suggest a route to the more modest goals of controlling walking direction and speed as well as making any major load adjustments.

Adaptive estimation

The relationship between the intended movement of a person with tetraplegia and the relatively tiny ensemble of neural signals used to deduce that intended movement is by no means expected to remain stationary. Although this relationship can be stable over time (31), both the technical factors (e.g., micromotion of the electrode with respect to the tissue, changing characteristics of recording devices) and biological ones (e.g., cortical plasticity) discussed earlier often change the mapping (111–113). Thus, adaptive decoders have been employed to refine continually the relationship between the recorded neural signals and the expected output. For example, using Bayesian regression methods, decoding performance was maintained in the face of model drift on the time scale of days (114).

Ultimately, the cause of the nonstationarities may arise from a range of cognitive or physiological state variables. For example, spikes rates may modulate by levels of attention. The amplitude of beta oscillations reduces upon initiation of movement (11) and the phasing of the rhythm has been shown to be predictive of spike activity (66). Thus, directly accounting for movement onset in the estimator may enable it to account for some of the nonstationarities. Similarly, incorporating eye position into the estimation process can raise iBCI performance (115). In this view, the observed nonstationarities more reflect the incomplete nature of existing models underlying estimators than changes to the random noise corrupting the neural signal.

Conclusion

Research progress in implants and decoders for iBCIs is advancing rapidly. 2006 saw the reporting of people with paralysis controlling a computer cursor and simple robotic devices. Only six years later, complex 3D reach and grasp movements of assistive robotic devices was achieved (see Figure 9) (25). Wireless, transcutaneous sensors have reached the milestone of initial successful animal testing (46). Furthermore, as has been the case in the neural prosthesis field for 50+ years, publicly funded preclinical research continues to provide vital insights and new directions for clinical investigation focused on developing restorative neurotechnologies for people with paralysis.

Figure 9.

Successful iBCI-based neural control of a robotic arm to reach and grasp a thermos of coffee by a person with tetraplegia. The MEA recording the neural data had been implanted for more than five years at the time of this achievement. (courtesy of (25))

At the same time, the number of decodable states is expanding (77) and other areas besides the arm are proving to be amenable to brain-control (110). The push for additional motor states is being supported by an expansion of the list of features extracted such as those seen with LFPs (61, 81). The developments promise to increase the capability of robotic aids or prostheses.

Just as important as the degrees of freedom is the quality of control. With the introduction of algorithms such as dimensionality reduction (79) and improved implant circuitry, the neural signal features are becoming less contaminated by noise. Noise free signals however will not cure errors in the model, but fortunately model sophistication is increasing as well (84, 99) and so are ways to avoid poor calibration (24, 25). These efforts fuel the hope of seeing finer control tasks performed in the coming years.

Finally, for the technology to be clinically viable, it would ideally be reliable for very long time periods, with a span of more than a decade being a plausible target. Preliminary reports, with one person successfully using an investigational iBCI (BrainGate) after 5 years, suggest great potential, however additional advances in sensor manufacturing will be essential (67). For example, the current work on a wireless, transcutaneous implant eliminates some connectors and tethering that can wear. When sensor characteristics change with time or the physiological mapping between intended movement and neural activity evolves, adaptive features built into the decoder can mitigate such deviations (114). These innovations all underscore the expectation that iBCI users will soon see implants functioning well for 10 years, and frequent cases where iBCIs are used for days at a time without any recalibration.

The iBCI field is tasked simultaneously with creating better devices for fundamental neuroscience research and with creating viable clinical neurotechnologies. This latter goal requires continued advances in neuroengineering, as well as rigorous clinical evaluation. With iBCI’s becoming more capable, precise, and reliable, future research will need to then bridge the gap between demonstrating a capability in a research trial and incorporating the technology into the daily lives of its users. For people who are unable to move or speak due to brainstem stroke or motor neuron disease, the impact of even modest increases in command signal output will be immense. While this and other goals for iBCIs will likely require years of additional research to become reality, recent strides make clear that such technologies are moving steadily from the realm of science fiction toward the domain of devices that support the communication, mobility, and independence of people with neurologic disease or injury.