Toward more versatile and intuitive cortical brain machine interfaces

Abstract

Brain machine interfaces have great potential in neuroprosthetic applications to assist patients with brain injury and neurodegenerative diseases. One type of BMI is a cortical motor prosthetic which is used to assist paralyzed subjects. Motor prosthetics to date have typically used the motor cortex as a source of neural signals for controlling external devices. The review will focus on several new topics in the arena of cortical prosthetics. These include using 1) recordings from cortical areas outside motor cortex; 2) local field potentials (LFPs) as a source of recorded signals; 3) somatosensory feedback for more dexterous control of robotics; and 4) new decoding methods that work in concert to form an ecology of decode algorithms. These new advances hold promise in greatly accelerating the applicability and ease of operation of motor prosthetics.

Introduction

A brain machine interface is a system that can interface the brain with computers and other electronics and can be used to assist subjects with neurological deficits. These devices can “write-in” signals, typically through electrical stimulation. Examples of write-in BMIs are cochlear implants for restoring hearing, retinal implants for restoring site. “Read-out” BMIs record brain activity, often at the level of action potentials from populations of neurons. A prime example of a read-out BMI is a cortical neural prosthetic used to assist paralyzed patients. This prosthetic records neural activity, decodes the subject's intent, and then uses this processed intention signal to control external devices such as a computer, robotic limb, or wheelchair.

In this review we will concentrate on cortical neuroprosthetics for motor control; i.e. prosthetics that assist paralyzed subjects. Motor prosthetics primarily rely on reading out neural activity but they can also write-in feedback signals such as somatosensory feedback. Motor prosthetics to date have typically used the motor cortex and have been previously reviewed [1-3]. In the current review we will focus on several new topics in the arena of cortical prosthetics. 1) Expanding the source of control signals to areas outside motor cortex: current motor prosthetics have used signals recorded primarily from the limb representation of motor cortex. This cortical region is close to the motor output and is concerned with the coding of trajectories of limb movements. Areas that carry the intent to make movements at a higher, cognitive level such as the posterior parietal cortex (PPC) may allow more intuitive and versatile control. Areas specialized for higher cognitive functions of speech and categorical decision-making may likewise be more adept at controlling a variety of devices from speech synthesizers to computer tablets. 2) Local field potentials (LFPs) as a source of recorded signals: whereas spikes are the standard signal for BMIs, LFPs can provide both complementary and supporting signals. A primary hurdle of current prosthetic recordings is the loss of spiking activity with time. LFPs, which are recorded from a larger “listening sphere” than spikes, may extend the lifetimes of array implants. 3) Somatosensory feedback for more dexterous control of robotics: current prosthetic applications in humans are only of the “read-out” variety and rely on vision for feedback. For patients with spinal cord and other lesions suffering from both paralysis and loss of somatosensation, writing-in somatosensory signals from electrical stimulation of cortex promises to provide an additional feedback pathway for improving prosthetic performance. 4) New decoding methods that work in concert with the decode algorithm to form a decoding ecology: these ecologies contain a mixture of components that optimize training, calibration, feature and parameter selection, match effector dynamics, and estimate intentions.

New cortical areas for motor prosthetics

Motor cortex has the attractive feature of being close to the motor output and thus can provide signals highly correlated with desired movement trajectories and the degrees-of-freedom of robotic limbs. However, many of the operations of an external device, such as a robotic limb, do not require all the details of a movement be derived from brain signals to provide dexterous operation. Smart robotics and computer vision can assist in executing the intent of the subjects determined by recordings from cortical areas upstream of the motor cortex to drive prosthetic devices [4]. Moreover, since the intent encoded in these upstream areas is more general than in motor cortex, they may be more flexible and intuitive for prosthetic operations [5, 6]. In this section areas of the posterior parietal cortex (parietal reach region, area 5, anterior intraparietal area, and lateral intraparietal area), the premotor cortex, and language areas will be discussed as candidate new targets for recording signals for prosthetic applications.

Posterior parietal cortex

The posterior parietal cortex (PPC), particularly regions around the intraparietal sulcus (IPS) and superior parietal lobule (SPL), provide a bridge between sensory areas in the caudal cortex and motor areas in more rostral cortex. The properties of neurons in this region cannot be classified as simply sensory or motor, but rather have properties of both and are involved in sensorimotor transformations.

Four general features of the PPC are particularly advantageous for prosthetic control. 1) The cells often encode the goal of movements. This goal specificity is advantageous for rapid operations [7] and for constraining decode algorithms when used in combination with trajectory signals which are also present in PPC [8]. 2) The representation of limb movements is bilateral [9, 10]. Whereas sensory areas represent the contralateral space, and motor cortex the contralateral limb, PPC represents both limbs and both spatial hemifields, and thus a single implant in one cerebral hemisphere can be used for decoding bimanual operations across space. 3) Sequences of movements are represented in PPC [11]. Most motor behaviors require a sequence of individual movements which can be read out at once in PPC. 4) LFP power is very strong in PPC [12]. The LFP signal can be used to improve decoding performance when combined with spiking activity and is often complementary to spiking activity [13]. It is possible that LFP recordings remain robust for longer periods of time than spike recordings. If this is the case, then LFPs recorded from PPC can be used to substantially extend the recording lifetime of implants, currently a major hurtle in the field.

Map of intentions in the PPC

The PPC is a source of intention signals which represent plans or potential plans for movement. Moreover, there is a topographic map of intentions within the PPC of both non-human primates (NHP) and humans [14, 15]. The reach representation consists of a complex of areas. Much of this complex was originally defined as the parietal reach region (PRR) and includes the medial bank of the IPS and the anterior bank of the parieto-occipital sulcus (POS). Subsequent studies have further divided these regions into V6a (in the POS), and MIP and area 5v in the medial IPS. Another part of this reach complex is dorsal Brodmann's area 5 (area 5d). Also in PPC there is a grasp specific area, the anterior intraparietal area (AIP), and a saccade specific area, the lateral intraparietal area (LIP).

Parietal reach region

The parietal reach region (PRR) was first described in NHP studies as a cortical region which is more active for reaches than saccades [16]. It strongly represents visual or auditory targets as goals for reaching, predominantly in eye-coordinates [17, 18]. The effector specificity for goals in PRR is bilateral, with some cells representing reaches with the contralateral limb, others for the ispilateral limb, and many cells for both limbs [9]. This area represents potential reach plans, prior to a decision, as well as the outcome of the decision [19]. The robust goal and decision making features should translate into PRR being able to specify a reach more quickly than motor cortex due to its rich source of information about the final goal of a movement. NHPs can use the planning activity in PRR for positioning a cursor on a computer screen under brain control [7]. PRR can also represent simultaneously two goals and their sequence in a sequential reach task [11]. Thus it is an excellent source of reach goal information for BMI applications.

Similar to NHPs, humans have a complex of reach selective areas in the PPC as demonstrated by fMRI, MEG, and TMS studies in healthy subjects and neuropsychological studies in patients with brain lesions. This complex includes two prominent regions, one located in the medial intraparietal sulcus, and the other more medially in partially overlapping regions referred to as the precuneous, parieto-occipital junction, and superior parietal occipital cortex (Figure 1) [20-26]. Lesions in these regions produce optic ataxia (OA) manifesting as inaccuracies in reach to visual targets in the periphery, largely in eye coordinates [27-29]. Inactivation of PRR in NHPs also produce OA [30], and as mentioned above, neurophysiological studies indicate that PRR represents reach targets in eye coordinates, thus giving further evidence for similarities between these regions in the two primate species.

Figure 1. Organization of intent in human posterior parietal cortex.

Areas are activated preferentially for reaching or pointing (red dots), grasping (blue dots), and eye movements (green dots). Adapted from [14].

A practical difficulty in using the above two candidates of human PRR for neuroprosthetic applications is that these regions are located largely within sulci. Current FDA-approved microelectrode arrays for human implant are the silicon-based arrays produced by Blackrock Microsystems. They are often referred to as “Utah” arrays since they were originally developed by Normann and colleagues at the University of Utah [31]. These arrays, like most silicon-based arrays, are of limited length and cannot reach these deeper areas. This issue of electrode lengths is important to address with neuroengineering and regulatory efforts since it is problematic for many cortical areas with intent signals. Longer silicon-based arrays and microwire-based arrays have been developed (for example [32, 33] but these longer arrays have not received FDA approval. Approximately 2/3s of the cortex is located in sulci; for instance, the hand representation of motor cortex is located deep within the central sulcus.

Area 5d

One of the areas of the reach complex is located on the gyral surface in both NHPs and humans, area 5d (Figure 1). It has the important feature of carrying signals related to both the goal of a movement and the trajectory. The trajectory signal, based on its timing and on deficits with lesions to this area, appears to be an efference copy of motor commands and may be used as a forward model for state estimation [34, 35]. Unlike PRR, area 5d only represents the movement plan that is the outcome of the decision and not potential plans [19]. Also it largely represents reach targets in limb coordinates [36].

Area 5d neurons have been used in BMI applications in which trajectory decoding is used to move a cursor on a computer screen by NHPs [8, 37]. Interestingly, this trajectory decoding can be accomplished without any actual limb movement and thus appears to represent the forward model of an imagined movement or gated motor command. The decoding was accomplished in 3D space and with or without eye fixation [37]. These studies show that area 5d is also a very good location for extracting control signals for neural prosthetics.

Over the last year we have implanted one human tetraplegic patient with a 96 electrode silicon-based “Utah” array (Blackrock Microsystems) in presumed area 5d based on fMRI localization of imagined reaches. The subject, over a period of recordings of one year, is able to control the 3D endpoint of cursors on a computer screen and the hand location of a robotic limb [38-40].

Anterior intraparietal area

The anterior intraparietal area (AIP) is a region in the anterior aspect of the IPS whose cells are selective for objects and the hand postures to grasp them [41-43]. Inactivation of AIP produces misshaping of the fingers during grasping [44]. In humans, fMRI and TMS studies have identified what appears to be a homologous area specialized for grasping within the anterior IPS at the junction with the postcentral sulcus (Figure 1) [21, 45, 46].

The same patient that we implanted with a Utah array in presumed area 5d also had an additional 96 electrode array implant in presumed human AIP [38-40]. Although in humans and non-human primates AIP is located primarily within the intraparietal sulcus, in humans a portion of AIP extends onto the gyral surface. We found that the cells recorded from the AIP array are selective for a variety of hand postures and these signals were used for online grasping of objects with a robot hand under brain control. A special feature of the AIP activity is that a number of grasp postures can be decoded using a small set of AIP neurons.

Lateral intraparietal area

The lateral intraparietal area (LIP) is located on the lateral wall of the posterior half of the IPS in NHPs. Similar to PRR, it codes goal locations in eye-centered coordinates, but for saccades rather than reaches [16]. While we are not always looking toward where we are reaching, the overall statistics of the patterns of eye-hand movements can be used to improve decoding of reach activity from PRR by providing an additional channel of information [47]. Thus recordings from LIP can be used to improve decoding of reach targets when combined with PRR activity during free gaze and eye-hand coordination. Area LIP neural populations provide an accurate on-line coding of both the direction of eye movements and current eye position [48]. Additionally, it has been shown that LIP activity can be used in a brain control task in which eye movements are planned but not executed [49]. Thus LIP can serve as a substitute to bulky eye monitoring equipment used for tetraplegic patients, and as a primary communication channel for locked-in subjects.

LIP has also been determined to be a primary site for learned categorization of visual stimuli [50, 51]. The categorization function pertains to a variety of stimulus attributes including motion direction and shape. Interestingly this categorization function is stronger and appears earlier when directly compared to the dorsal prefrontal cortex, an area previously implicated in categorization [52]. It also appears more robust than categorization-related activity in inferotemporal cortex. Thus LIP is a possible site for using a computer tablet where the desired visual icon is directly categorized and decoded. This direct method would bypass the need for move mouse and click operations for tablet use.

A presumed homology of LIP has been identified in humans on the medial bank in the middle of the IPS (Figure 1) [53].

Premotor areas

Similar to PPC, there are a number of areas in premotor cortex that can serve as sources of signals for neuroprothetics. Similar to PPC, the premotor cortex also contains a map of intentions with the dorsal premotor cortex (PMd) selective for reach, ventral premotor cortex (PMv) for grasp, and the frontal eye fields (FEF) for saccades. Not surprisingly the predominant reciprocal parieto-frontal connections of these areas are with PPC areas showing similar selectivity; i.e. PMd with PRR and Area 5d, PMv with AIP, and FEF with LIP [42, 54-57]. PMd neurons in NHPs have been used as a source of goal signals in BMI studies using brain control of a computer cursor [7].

Language areas

Dysarthria, poor articulation, or anarthria, a complete loss of speech, can occur with adult and childhood diseases. Adult language defects that potentially could be amenable to BMI speech assistance include those conditions in which the patients are “locked-in” as seen with amyotrophic lateral sclerosis (ALS) and brainstem strokes [58, 59]. Childhood diseases that affect speech include Duchenne muscular dystrophy and cerebral palsy.

In healthy individuals, speech production activates a complex network of brain structures, including the superior longitudinal and arcuate fasciculi, insular cortex, supplementary motor cortex, basal ganglia, cerebellum, Broca's area, Wernicke's area, and motor face and pre-motor cortex [60-62]. Thus, speech BMIs could read out both cognitive and motor signals to provide fast, intuitive speech output [59, 61, 63-65]. Importantly, critical parts of the speech network actuate during both overt and covert speech production [62, 66], suggesting that their activity may be available even after the onset of the locked-in state [67].

Local field potentials in the Cerebral Cortex

From action potentials of single neurons to scalp potentials averaging many thousands of neurons, widely different scales of neural activity have been used to operate cortically-driven prostheses [68-73]. Selecting the signal for a brain-machine interface requires balancing the need for high fidelity in the recorded brain signals against the risks of electrode placement. Where applications justify the risk of intracortical implantation, single-unit activity is often the highest-priority signal since action potentials directly represent cortical information processing. However, the local field potential (LFP), which may be recorded simultaneously with action potentials, represents another important signal. In addition to supporting BMI applications [68], the value of the LFP is underscored by its increased utilization in both clinical and academic research [74-79]. Furthermore, the LFP is a durable signal, persisting after isolatable action potentials are no longer available [68, 80, 81].

Intracortical LFPs are mesoscale brain signals representing, to a large degree, ionic membrane processes summed over local populations of neurons [82, 83]. These processes include synaptic currents, calcium spikes, intrinsic currents, afterhyperpolarizations, gap-junctions, and neuron-glia interactions [82, 84]. Additionally, action potentials can contribute to the upper frequency range of the LFP when fired in synchrony across a population of neurons [82]. The magnitude and polarity of these contributing processes, and the synchrony between them, contribute to the strength of the potential. The spatial extent of LFPs has been estimated to be as small as a few hundred micrometers [85-87] or as large as several millimeters [88-90], with dependencies on the brain area, species, cytoarchitecture, brain state, task, synchrony of inputs, and frequency bands (Figure 2). Field potentials may also be recorded at the cortical surface (electrocorticography, or ECoG) or at the scalp (electroencephalography, or EEG). ECoG recorded with millimeter-scale electrodes can resolve millimeter-scale spatial dynamics [59, 91], whereas EEG typically has a spatial resolution in the centimeter range.

Figure 2. Scales of neural activity.

The dynamics of neural signals can be recorded at diverse scales, from single neurons to heavily averaged scalp recordings. The scale of the recording should match the requirements of the application. The local field potential, particularly when recorded in a grid of tightly-spaced electrodes, can accurately reflect the dynamics of the surrounding cortical population activity with high spatiotemporal fidelity. Legend: EEG (electroencephalography), ECoG (electrocorticography), uECoG (micro-electrocorticography), LFP (local field potential), AP (action potential).

Field potentials in cortical information processing

The LFP is a rich source of information about cortical processing. Much of this information is distributed among distinct frequency bands with varying spatiotemporal characteristics. In the frequency domain, spectral power of field potentials falls as 1/f^χ, with χ ≈ 1-2, and spatial extent tends to decrease with increasing frequency [92-94]. Some of this falloff occurs because of the capacitive filtering of bi-lipid cell membranes, which attenuate high-frequency dendritic oscillations near the soma [82]. There is some debate whether the extracellular medium may also have a filtering effect, although recent studies have suggested that it is largely ohmic in the frequency range of the LFP [95, 96]. Frequency dependence may also be a reflection of the underlying network in that the limited spatial reach and lower amplitudes of higher frequencies suggests smaller contributing neuronal populations [82]. These network-level observations are further strengthened by evidence that gamma-band LFP signals often exhibit task-related power augmentation more tightly coupled to the underlying topography, whereas lower frequency signals tend to persist over larger spatial areas and depict state more than action [76, 93, 97-101]. In fact, spectral power of the higher frequencies tends to be modulated by the phase of the lower frequencies, a phenomenon known as phase-amplitude coupling [102, 103]. Very high frequencies in the LFP may also correlate with multi-unit activity, that is, the activity of nearby neurons whose waveforms cannot be isolated from the background noise [104]. These characteristics hint at the complexity and richness of the LFP signal, and can be used to better understand how the LFP might reflect cortical information processing.

It has not been conclusively determined whether the LFP is an epiphenomenon of neuronal activity or whether it might play an active role in cortical processing. One way in which the LFP may shape neuronal activity is as a feedback mechanism: field potentials modulate membrane voltages (a process known as ephaptic coupling), thereby influencing the timing of action potential outputs [105-107]. Others have suggested that spike timing depends on oscillatory phase coupling between distal and proximal brain areas [108]. Indeed, synchrony among cortical elements, often a result of network oscillations, has been implicated as a means of coordinating the activity of neuronal populations to produce coherent outputs [97, 100, 108-110]. However, the LFP and neuronal spiking activities do not always oscillate in lockstep; studies examining relationships between these signals and the blood oxygen level-dependent (BOLD) response have found that gamma-band (roughly, 30-80 Hz) power is a better predictor of the BOLD response than spiking rates, and have even shown that gamma-band power could be dissociated almost entirely from spiking activity [111, 112]. In practice, LFPs have been shown to contribute information complementary to neuronal spiking [13]. Although the contribution of the LFP to cortical processing is not fully understood, LFPs may be interesting for BMI applications both because of the way they reflect and may influence cortical information processing, and because they may complement the information available from simultaneously recorded action potentials.

Utility of LFPs for cognitive BMIs

Field potentials from the cerebral cortex have been shown to encode quantities as diverse as the functional areas from which they were recorded. For example, intracortical LFPs from motor cortex have been used to decode three-dimensional arm and hand trajectories [113-116], target direction and movement direction [117, 118], and elements of language, including vowels, consonants, or whole words from speech-related areas of cortex [59, 66].

The extension of the decoding of kinematics is the decoding of goals – an important cognitive distinction since goals are a more abstract quantity than movements. Most work exploring goal decoding has used signals recorded from the parietal cortex, an area of the brain heavily involved sensorimotor integration and high-level motor planning [15]. Hwang et al. showed that LFPs recorded from parietal cortex could be used as a “go” signal for movement execution in a BMI task [99], and that LFPs recorded from the parietal reach region (PRR) were more informative than spikes when comparing single channels (Figure 3a) [13]. Scherberger et al. additionally illustrated how the temporal structure of LFPs from parietal cortex vary with different types of planned or executed motor behavior, and that the LFP signals were predictive of the direction of the currently planned movement in single trials [101]. Fewer LFP channels were required for predicting behavioral state (Figure 3b). Thus LFPs can provide complimentary information to spikes, i.e. cognitive state. However, LFPs can also provide support for reach direction decoding when combined with spikes by providing additional channels of information [13].

Figure 3. Decoding of direction and behavioral state from LFPs.

A) Average direction decoding for each number of simultaneously recorded channels for spikes, LFPs and spikes and LFPs combined. B) Decoding error for behavioral state as a function of the number of recordings for spikes (black) and LFPs (red). The behavioral states are indicated by the symbols: baseline (circles), reach planning (squares) reach execution (diamonds), saccade planning (upward triangles), and saccade execution (downward triangles).

The qualities of the LFP make it an intriguing possibility for use in cognitive BMIs [13, 68]. Importantly, there is evidence that individual subjects can be trained to modulate power in different frequency bands of the local field potential [99, 119]. State variables such as movement onset execution, selective attention, and planning all play important roles in the cognitive load of behavioral outputs. Because LFPs can encode both global state variables [87, 99, 120] as well as active execution variables such as trajectory velocity and goal locations [13, 59, 66, 113-118], these signals can benefit both cognitive and behavioral components of BMIs.

Somatosensory feedback

Imagine you had to make a terrible choice: either you lose your eyesight or your sense of touch. What would you choose? Most people value vision as the most important sense. This is no surprise considering how much neural circuitry of primates is devoted to processing visual information [121]. The role of the sense of touch, however, is typically underestimated. One reason for appreciating touch less than the visual and auditory modalities might be that the classical sense of ‘touch’ actually comprises several sensory systems which are seldom perceived separately and almost never completely fail. Tactile perception, thermoception (perception of temperature), nociception (perception of pain) and proprioception (perception of joint angles) are often referred to as ‘somesthetic senses’. A partial or complete loss of the somesthetic senses can have dramatic effects on the quality of life. One very well documented case of a complete loss of the sense of touch is the case of Ian Waterman. Even with visual feedback agility is greatly reduced and controlling movements requires utmost concentration. “It's like having to do a marathon everyday”, Mr. Waterman aptly describes it according to Cole's book [122] (p. 169). If moving your own body without somesthetic feedback is such a hard task, how difficult must it be to control an extracorporeal prosthesis that lacks this kind of feedback?

Bidirectional brain machine interface

Most current neuroprostheses are unidirectional and feedback can only be obtained from vision. They can be effective [123-125] but the addition of somatosensation has great potential and could overcome some of their still existing limitations. Sometimes the missing somatosensory channel can be substituted by a different modality like vision [73, 123, 124, 126-135], mechanical stimulation of other body parts [136], or targeted muscle reinnervation [137]. Most of these options are not available to a tetraplegic or locked-in patient. Sensory substitution might provide some help but cannot provide detailed and intuitive feedback. Another possibility is to bypass the spinal cord and directly stimulate the somatosensory cortex using intracortical microstimulation (ICMS) (Figure 4). Wilder Penfield popularized the concept of a cortical homunculus – a representation of the whole body surface organized topographically on the cortex – in the first half of the 20^th century (Figure 5a) [138, 139]. By epicortical stimulation of the motor and somatosensory cortex during epilepsy surgeries he mapped the surface of the primary motor and somatosensory cortices to their corresponding body parts. Body parts which are more densely innervated occupy correspondingly larger areas on the cortex. Stimulating the appropriate part of the sensory homunculus while the corresponding part of a prosthesis touches an object could accordingly be interpreted by the brain in a similar fashion as if one's own limb touches the object. This forms the basic idea of a bidirectional brain machine interface – not only does the interface read-out cortical signals related to intended movements, but also writes-in somatosensory feedback (Figure 4).

Figure 4. Schematic overview of a bidirectional brain machine interface.

Tactile sensory information (red) flows from the prosthesis (left) through a computer and is converted into stimulation pulses which reach the brain (right). At the same time LFP and spiking activity is recorded and processed to generate a control signal for the prosthesis (blue).

Figure 5. Organization of somatosensory cortex.

A) Schematic depiction of the somatotopic organization of the primary somatosensory cortex. The body is represented ‘upside down’ from medial to lateral. Figure modified after [140]. B) Rostral to caudal organization of the somatosensory complex in the owl monkey. The inset shows a general overview of the owl monkey brain (bottom left) and the flatted representation (right) shows details about the subdivision into area 3a, 3b, 1 and 2. Primary visual and motor cortex and other areas are shown as reference. Red lines show folding lines for the flattened cortical map. Figure modified after [141].

The somatosensory cortex

The target for providing ICMS feedback is the primary somatosensory cortex (S1), consisting of Brodmann areas 1 (BA 1), 2 (BA 2), 3 (BA 3; further subdivided in 3a and 3b), which lies in the posterior bank of the central sulcus and surface of the postcentral gyrus (Figure 5a). From anterior to posterior the areas are ordered 3a, 3b, 1 and 2 (Figure 5b). 3a receives primarily proprioceptive information from muscle spindles [142] and has also been shown to be activated by thermal and painful stimuli [143-145], it projects to 3b, BA1, BA2 and the supplementary motor area (SMA). 3b mainly receives cutaneous input and projects to BA1 and BA2 [146, 147]. BA1 receives input from BA2, 3b and 3a, is activated by cutaneous stimulation and sends feedback to 3b [147]. BA2 has some sensitivity to cutaneous stimuli but is also activated by deep receptors, like muscle spindles [147]. The S1 complex contains multiple somatotopic maps (each subregion contains a full somototopic map of the body) which correspond to areas on the skin from the contralateral side. The somatotopic representation of different parts of the body follows a medial to lateral order in which the feet are represented first (most medial) and the face last (most lateral) (Figure 5a). This order mirrors that of the primary motor cortex (M1). It is therefore necessary to target different parts of S1 depending on the type of prosthesis to be used. Electrode arrays are relatively small compared to cortical limb representations in humans and parts of S1 lie within the central sulcus and are inaccessible with current array electrode technology. Thus finding the right implantation site becomes very important.

Stimulating the brain

Brain stimulation has been done routinely in surgical settings for over 80 years. Large electrodes with a diameter of a few millimeters are typically used to determine functional properties of parts of the brain prior to epileptic surgery. Large electrodes and high electrical currents activate large areas of cortex and correspondingly have a low spatial resolution. For chronic implantation, devices have to be much smaller and electrical currents much lower to provide high precision. One technology that is in clinical testing at the moment is the Utah electrode array (Blackrock Microsystems) consisting of 96 recording electrodes arranged in a 10 by 10 grid which covers an area of 4 by 4 mm. The electrode tips are treated with a sputtered iridium oxide film (SIROF) to lower the impedance and thereby further improve their stimulation performance [148]. Chronically implanted, these arrays function over long periods of time [149-151]. They can produce a more precise stimulation than contact electrodes [152], enabling a high spatial resolution. Even the small electrodes used for DBS (electrode diameter = 1.27 mm) are huge compared to cortical microelectrodes (electrode diameter at tip 2-3 μm) which could be used to stimulate the somatosensory cortex. The smaller size of the electrodes also has the advantage to produce less trauma when inserted into the brain.

Another important factor is the electrical current itself. Direct stimulation of the brain has the risk of causing tissue damage [152-155] and can potentially induce seizures [156]. Therefore the stimulation parameters of frequency, amplitude and duration have to be carefully chosen to prevent dangerous side effects. A recent NHP study has explored the safe and effective parameter space for a neuroprosthetic application [157]. Varying frequency and amplitude within these safe margins may allow simulating the variable force of a touch or the texture of objects [150, 158, 159].

Artificial sensations

How does it feel when the somatosensory cortex is directly stimulated? This question is hard to answer since no systematic human subject trials which utilize stimulation of the somatosensory cortex to create artificial sensations have yet been started to this date. The sensation will likely be of very artificial nature in a similar way as sensations elicited by stimulating the visual cortex [160-162], the cochlea [163-165] or peripheral nerves [166-169]. Through plasticity these new sensations may over time be learned and integrated into the body schema in a similar manner as cochlear implant recipients learn to interpret sensations from their implants. What is known from somatosensory cortex stimulation studies in NHPs is that the stimulation can be detected and different stimulation parameters can be distinguished from others [150]. In a study by Berg et al [170] NHPs were able to distinguish varying ICMS amplitudes that were used to simulate varying physical indentations of a finger. The animals were as good in discriminating variable physical indentations as they were in discriminating corresponding artificial stimulation amplitudes which indicates that they might have perceived both types of stimuli similarly. Another study which combined brain control and stimulation showed that NHPs could discriminate two different stimulation frequencies [159]. Varying only these two parameters – amplitude and frequency – may be enough to recreate a large variety of tactile sensations.

Stimulation challenges

Having sensitive recording electrodes and stimulation electrodes in close proximity in the same electrically conductive tissue produces considerable stimulation artifacts. If microelectrodes in the somatosensory cortex are stimulated, the signal is picked up by the recording electrodes used for controlling the neuroprosthesis. Several approaches can be made to avoid this interference. One solution is to stop recording data during stimulation, which is not very viable since it does not allow continuous feedback control of a bidirectional interface. Another possibility is to filter out the artifact or reconstruct the original signal. This approach has been used successfully for EEG signals that were contaminated by transcranial magnetic stimulation [171]. In a current study from our lab we applied a similar filtering approach for artifact removal during online brain control of a cursor [172]. Although the stimulation artifact is picked up by recording arrays in other brain areas it is also possible that neurons are directly activated at the recording site from orthodromic or antidromic conduction from the stimulating site. However, neither our study nor a similar earlier study [159] reported such activation or its interference with decoding.

Decoding ecosystems

A successful BMI allows the user to perform useful actions in their environment. For over a decade, improved implementations of BMIs have seen the quality of closed-loop cortical control improve from relatively crude proof-of-concept demonstrations to control that is beginning to approximate that of natural limb movements [71, 126]. These improvements cannot be attributed to any single factor but instead are the consequence of the evolution of an ecosystem of methods that work together to enable high-performance cortical prostheses (Figure 6).

Figure 6. Schematic of a decoding ecosystem.

The successful implementation of a brain machine interface relies on the development of a decoding ecosystem. The decoding ecosystem not only defines the algorithm used to translate neural activity into effector actions, but also the process through which the parameters of the decoding algorithm are derived and constrained to optimize online control. This figure highlights seven components of a decoding ecosystem. Parameter Selection: Selection of the type(s) of information that will be decoded from the neural population (e.g. position, velocity, goal.) Calibration Task: The task used to decipher the relationship between neural activity and the subjects intentions. Training Protocol: The protocol specifying how a decoder is initially configured and subsequently updated based on the user's attempted neural control. Assistance Methods: Methods used to assist the subject in attaining task goals during neural control. This can be used as a bridge that helps the subject achieve full unassisted control as part of the training protocol or can be used to encourage acquisition of neuroprosthetic skill. Intention Estimation: Methods to model the user's instantaneous motor intentions based on noisy decoder output and known task goals enabling improved decoder calibration using data acquired during attempted neural control. Feature Selection: Procedures for sub-selecting relevant neural channels to prevent overfitting and/or remove channels that correlate with subject intentions but do not positively contribute to online performance. Effector Dynamics: Methods that tailor the dynamics of the prosthetic effector to ongoing task demands.

Algorithms versus ecosystems

The decoding algorithm is the mathematical method used to transform neural activity into actions. A natural assumption is that improvements in online control must be the consequence of improvements in the decoding algorithms used to interpret neural activity. Many decoding algorithms have been used for closed-loop cortical control [e.g. 71, 125, 173, 174]. While between-study comparisons are nearly impossible given differences in task design, training paradigms, performance metrics, quality of the recorded neural population, and implementation details, within-study comparisons of decoding algorithms have demonstrated the importance of decoder selection [71, 173, 175].

However, no algorithm operates in a vacuum (Figure 6). Certain practical considerations can affect how the decoder performs online. For instance, ideally, neural activity should influence decoder output in proportion to how much independent information it carries about the decoded variable. Unfortunately, decoding methods that leverage this idea can hurt online control if enough training data is not available to estimate channel independence (training protocol, Figure 6). Acquiring more training data would be one solution, however, this comes with a cost – the user of the BMI must spend more time calibrating and less time using the decoder.

Other considerations include the tasks used to decipher the functional properties of the neural population, how task data is analyzed to parameterize the decoder, how neural activity is processed into the features that are used by the decoder, what types of information should be extracted from the neural population (e.g. position, velocity, goal), and how to ensure the dynamics of decoder output are suited for task objectives (see Figure 6). Decisions regarding these factors can affect the ability of the subject to utilize the BMI as much as the selection of the decoding algorithm. Additionally, as the example of attempting to leverage neural independence illustrates, choices made while implementing a BMI are not independent. For these reasons, it is important to think of implementations of BMIs as constructing a decoding ecosystem - a network of interdependent methods and choices that work together to enable high performance control (see Figure 6). The remainder of this section will highlight two key components of the decoding ecosystem: how neural activity is characterized for use in the decoding algorithm and how traditional methods from estimation theory can be modified to improve online control.

How neural information is characterized: correlations become causation

The decoding algorithm is parameterized by fitting neural data acquired during a calibration task to aspects of effector motion such as position and velocity. Importantly, the correlations observed during task performance come to define the causal transformation of neural activity into effector motion – correlation becomes causation. If the correlations recovered from the task are not an accurate reflection of how well the units will enable control, the performance of the BMI will suffer. Over 50 years of neural recordings in primary motor cortex (M1) have demonstrated the complex context-dependent properties of neurons in seemingly “simple” cortical areas. For instance, the direction tuning properties of M1 neurons [176, 177] are context dependent [178-180], are modulated by both visual and somatosensory information about the state of the limb [181-188], and are modulated by visual cues that inform movement [189-191]. Given such diverse and task dependent tuning of M1 cells, it is not surprising that the behavior of neurons during initial decoder calibration tasks and the behavior of neurons during closed-loop brain control can change [123, 125, 128, 192-194].

A key innovation in motor prostheses was to understand parameterization of the decoder as a process of successive iterations designed to reveal how neurons behave in the context of actual prosthetic control (training protocol, Figure 6) [124, 125]. This process begins with the subject attempting control using a decoder trained on observed effector movements [73, 126, 194]. If initial performance is poor, computer assistance can help the subject complete the task [126, 127, 132] similar to the use of interactive robotic therapy for neurorehabilitation following stroke [195]. The data generated during the control session is used to construct a new decoder. The new decoder enables improved control because the implicit model of neural behavior that is codified into the decoding algorithm more accurately reflects the behavior of neurons during brain control [125, 194, 196].

The details of task construction and how task data is used to construct the decoder strongly affect ultimate decoder performance (see Figure 6). Recent advances attempt to better identify neural activity useful for control by fitting neural activity to the moment-to-moment intentions of the user as estimated from effector kinematics and known task goals (intent estimation, Figure 6) [71, 194, 196, 197]. Further details, such as the task that is used for calibration [128, 198] feature selection [126, 175, 196] selection of the parameter to decode [41, 71, 174], shifts introduced between kinematics and neural activity [199], and the protocol for how frequently decoder parameters are updated [200] also affect decoder performance. The decoding algorithm transforms correlations observed during training sessions into the causal mapping of neural activity into actions. A key feature of high-performance neural prosthetics is the protocol used to shape the patterns of correlations to optimize control.

Modifications of estimation methods to improve on-line control

The role of the decoding algorithm in neural prosthetics is unique. Unlike standard uses of decoding algorithms for open-loop estimation, the user of the BMI is in the loop – the user can modulate their neural activity to compensate for observed errors, and if the errors are persistent through time, they can learn to produce outputs that reduce anticipated errors [193, 201-204]. For these reasons, open-loop testing can often misinform the design of closed-loop decoding algorithms [175, 205, 206]. For instance, while open-loop decoding performance was optimized when neural data was integrated into windows of 100-300 ms, online testing showed that shortened integration windows of 25-50 ms were best for online control [206]. More generally, for closed-loop control, it is necessary to balance two competing objectives: minimizing phase-delays (roughly the amount of delay it takes for neural activity to cause effector motion) against minimizing jerk (the high-frequency oscillations of the effector) (effector dynamics, Figure 6). The trade-off between phase-delays and jerk is task dependent. If the operator of the BMI intends to make slow precise movements, effector jerk becomes debilitating while phase-delays are hardly noticeable. Conversely, if the intention is to make fast ballistic movements, phase-delays become debilitating while jerk is less noticeable. Intelligent optimization of key aspects of control that are sensitive to behavioral demands is an important part of parameterizing a decoding algorithm.

Recognizing the underlying representations of the neural populations is also critical for online control. For example, Kakei et al. [207] showed that populations of M1 neurons code movements in different reference frames with some neurons coding movement in an “intrinsic-space” (preferred direction, the direction of movement giving the largest neural response, defined with respect to the fore-arm) while other neurons code movement in an “extrinsic-space” (the preferred direction defined with respect to direction of action in space.) In the context of BMIs, correct interpretation of neural activity would depend on 1), tasks that are designed to dissociate the reference frame of units in the population, and, 2), use of non-linear decoding algorithms that can combine the mixed movement representations. An alternate approach could be to use more cognitive brain areas that encode movements in a simpler manner. In a companion to the above study, Kakei et al. [208] showed that neurons in premotor cortex encode movements almost exclusively in an extrinsic reference frame. This simple and homogenous representation of movements would simplify the tasks necessary to characterize neural behavior and allow the use of simpler decoding algorithms. This example not only highlights the potential advantages of cognitive neural prosthetics but also underscores the necessity of thinking of decoding in terms of ecosystems. Dependent on underlying neural representations, different task designs and decoding algorithms become necessary to optimize the amount of information that can be extracted for use by the BMI.

Conclusion

This review has described several new topics for motor-cortical prostheses: using cortical areas outside of motor cortex; using LFPs as a source of information; using intracortical stimulation to “write-in” information back to the brain; and shifting perspective on decoding from independently operating algorithms to ecosystems, of which the algorithm is just one essential part. Each of these concepts is an important consideration in designing more intuitive and versatile BMIs.

High level cognitive signals recorded from areas beyond motor cortex are just now being used for motor prosthetics [38-40]. The intent signals from PPC are versatile in controlling a number of output devices. The cognitive signals can be used in combination with smart robotics to interpret the intent of the subject without the high overhead of brain control of all of the degrees of freedom of the limb. The same areas that can control robotics can also control a computer tablet and other computer interfaces. Moreover, the point and click operations may be replaced by direct decoding of the targeted icons. This increased intuitiveness of control can be extended to other high level functions by choosing implant sites that naturally encode the desired functions. For instance, implants in language areas can allow speech to be directly decoded without having to go through additional steps, e.g. spell boards on computers.

Local field potentials may be recorded simultaneously with action potentials, and may be more robustly available over the lifetime of the recording electrodes than action potentials. LFPs reflect the activity of local populations of neurons, and can encode both state and execution variables.

The dexterity of robotic hands can be greatly improved by integrating neural decoding and somatosensory cortex stimulation into an effective bidirectional BMI. Two studies are about to start which will involve online brain control while simultaneously giving sensory feedback via ICMS [209]. These ground breaking studies will not only determine for the first time what artificial stimulation actually feels like but will likely also improve manual dexterity which will lead to an increased quality of life for the patients.

Advances in the decoding ecosystem have led to dramatic improvements in the quality of neural prosthetic control. The hope is that recent successes in laboratory settings will give rise to more widespread clinical testing and eventually adoption of BMIs as a method to enhance the quality of life of the affected clinical population. The use of BMIs in richer environments with more diverse contexts will bring new challenges if scientific studies of neural behavior during natural motor control extend to neural behavior during the operation of BMIs.

Although this review covered four topics—new areas, LFPs, somatosensory stimulation, and decode ecologies—there are a number of intersections between these topics, particularly in regard to future developments. 1) The implantation of new areas will uncover new LFP characteristics that, using LFP oriented components of the decode ecology, will advance performance. One current example is the use of PPC as a source of signals; the PPC has very high LFP power that allows robust state decoding. 2) Another example of an intersection between areas and LFPs is the recent implantation of two cortical areas, AIP and area 5, in a human patient. The use of two areas opens new frontiers for novel signals and decoding methods [38-40]. Animal studies have shown context dependent flow of information between cortical areas that can be traced by partial spike field coherence analysis [12, 210]. The spiking in one area, correlated with the LFPs in another after removal of potential common drivers, provides insights into how information is being communicated between cortical areas. Such measures of partial spike-field coherence can provide an entirely new window in understanding the context and intent of the subject. 3) Somatosensory stimulation feedback will require new components of the decode ecology to filter out artifacts and to make on-line corrections by the subject based on this new channel of sensory feedback. 4) The decode ecology has been considered as a central algorithm assisted by a variety of processing components. Future extensions of the ecology will in many cases contain more than one decode algorithm. One current example is a decoding algorithm for directions of intended movements that is assisted by a second algorithm that decodes the state of the subject [211]. These two algorithms improve cursor control by stabilizing the cursor during periods when no movements are intended. 5) Moving to new areas will require new decode ecologies. This will be particularly the case when high level cognitive signals of language or categorization are to be decoded.

In conclusion, the field of neural prosthetics is moving forward on a number of fronts. These efforts will not only help patients, but will also expand basic science knowledge of the functions of the human cortex. These scientific advances will be achieved with the recording from different high level cortical areas, the combined use of different cortical potentials, and the development of algorithms that can provide insight into adaptability and encoding of information in neural populations. These new scientific findings will in turn provide insights into improving neural prosthetics. The future is bright for cortical neural prosthetics.