Biomimetic Brain Machine Interfaces for the Control of Movement

Abstract

Quite recently, it has become possible to use signals recorded simultaneously from large numbers of cortical neurons for real-time control. Such brain machine interfaces (BMIs) have allowed animal subjects and human patients to control the position of a computer cursor or robotic limb under the guidance of visual feedback. Although impressive, such approaches essentially ignore the dynamics of the musculoskeletal system, and they lack potentially critical somatosensory feedback. In this mini-symposium, we will initiate a discussion of systems that more nearly mimic the control of natural limb movement. The work that we will describe is based on fundamental observations of sensorimotor physiology that have inspired novel BMI approaches. We will focus on what we consider to be three of the most important new directions for BMI development related to the control of movement. (1) We will present alternative methods for building decoders, including structured, nonlinear models, the explicit incorporation of limb state information, and novel approaches to the development of decoders for paralyzed subjects unable to generate an output signal. (2) We will describe the real-time prediction of dynamical signals, including joint torque, force, and EMG, and the real-time control of physical plants with dynamics like that of the real limb. (3) We will discuss critical factors that must be considered to incorporate somatosensory feedback to the BMI user, including its potential benefits, the differing representations of sensation and perception across cortical areas, and the changes in the cortical representation of tactile events after spinal injury.

Introduction

Recording the activity of individual neurons within the brain of a behaving animal has been possible for the past 40 years, beginning with the work of Jasper and Evarts (Jasper et al., 1958; Evarts, 1966). Recently, however, it has become possible to record from hundreds of neurons simultaneously. By thus harnessing the natural control signals of the brain, rat (Chapin et al., 1999), monkey (Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003), and even human (Kennedy and Bakay, 1998; Hochberg et al., 2006) subjects have been able to control the movements of real and virtual objects directly from their thoughts.

For the most part, these brain machine interfaces (BMIs) build either directly or indirectly on the work of Georgopoulos et al. (1982), who discovered that movement-related discharge in the motor cortex (M1) depends on movement “kinematics,” or the trajectory of the hand through space (Georgopoulos et al., 1982). However, other work, including that of Evarts, suggests that the activity of M1 neurons is also related to movement “kinetics,” or the force-related variables that ultimately cause movement (Smith et al., 1975; Hepp-Reymond et al., 1978; Thach, 1978; Cheney and Fetz, 1980; Kalaska et al., 1989; Pohlmeyer et al., 2003; Morrow et al., 2007). The consequences of using neural signals with these latter properties for BMI control have been essentially unexplored.

Existing BMIs use only visual feedback to control movement, lacking the critical somatosensory feedback from sensory receptors in the muscles, skin, and joints of the moving limb. The movements of patients who lack this feedback are dramatically impoverished, even when muscle strength remains normal (Sanes et al., 1985; Sainburg et al., 1993; Abbott, 2006). In this symposium, we will examine novel BMI approaches based on kinetic signals and the potentially important role of somatosensory feedback.

The efferent interface: control of movement with kinetic commands

Prediction of joint torque

In one approach to using the signals of the brain for BMI control, one simultaneously records signals from both a large group of neurons and sensors that monitor the actual movements made by the subject. With these data, one can calculate a transfer function from the neural inputs to the movement outputs. In BMI parlance, this transfer function is typically referred to as a “decoder.” Using this decoder, it is possible to take a new set of neural recordings from which the subject's movements can be predicted. This step of decoder verification is called “cross-validation” (Stone, 1974; Browne, 2000). The accuracy of these predictions can be determined by comparison with the actual movements, for example, by calculation of the fraction of the movement variance that can be accounted for (FVAF) by the prediction (Serruya et al., 2003). If the predictions are done in real time, they can actually be used as control signals for a robot or computer cursor. By watching these movements, the subject can attempt to correct errors and guide the evolving movement. Nearly all such BMI experiments to date have predicted and controlled kinematic signals.

In one recent experiment, however, monkey subjects learned to track a moving target while controlling both the position and the force with which they gripped a joystick. After the monkey learned the normal task, control signals were taken directly from neural recordings rather than the joystick, using methods essentially like those described above. Ultimately, the force predictions based on M1 neurons proved to be more accurate than those of hand position (Carmena et al., 2003).

In this symposium, Hatsopoulos and Fagg will describe similar studies that predicted kinetic and kinematic signals related to the proximal limb. The monkey's arm was placed in a planar exoskeleton (KINARM; BKIN Technologies, Kingston, Ontario, Canada) (Fig. 1B) that allowed shoulder and elbow joint torque to be estimated from movement kinematics. Across six different datasets from three monkeys, performance in torque prediction was comparable with hand position prediction (mean FVAF of 0.61 vs 0.65, respectively, in which FAVF of 1.0 corresponds to a perfect prediction). Figure 2, A and B, compares one example of predicted (blue) and actual (red) movement. The result is encouraging, in particular given the greater complexity of the torque signals. Many questions remain to be explored, including how well the two approaches generalize to movements and conditions that differ from those used to compute the decoders.

Figure 1.

Experimental setup for motor BMI experiments. A, Recordings from chronically implanted electrode array are decoded to provide real-time control signals. B, In the Hatsopoulos laboratory, these signals can be used to control cursor position (gray circle at monkey's hand) either directly or indirectly through joint torque predictions acting on a forward dynamic limb model. In some experiments, the commands are also used to control the position of the monkey's own limb via the KINARM exoskeleton and thereby supply proprioceptive feedback to the monkey. Botox injections are being investigated as a means to reduce the monkey's ability to resist these imposed movements. C, An analogous system in Miller's laboratory involves a monkey trained to perform a wrist flexion task whose forearm flexor muscles can be temporarily paralyzed by injecting Lidocaine into cuffs surrounding the median and ulnar nerves. EMG predictions are sent to a stimulator and a set of intramuscular electrodes implanted in the paralyzed muscles. This allows the monkey to generate voluntarily controlled muscle contractions despite the paralysis.

Figure 2.

Movement-related signals predicted using neural discharge recorded from multielectrode arrays chronically implanted in the primary motor cortex. A, X-component of actual (red) and predicted (blue) Cartesian hand position during a sequence of movements between randomly positioned targets. Dashed lines indicate breaks between sets of movements. B, Shoulder torque predicted during the same movements. Torque prediction accuracy increased further with the addition of limb state (angular position and velocity; black lines) inputs. C, EMG signals predicted during a sequence of unconstrained reach and grasp movements. Flx Dig, Flexor digitorum; Ext Dig, extensor digitorum.

Fagg and his group have also tested more complex torque predictors that included inputs related to limb state, meant to simulate proprioceptive feedback. Performance of these models was significantly better than those based only on cell activity (p ≪ 0.001, paired t test), with an average increase of 0.11 in FVAF. Figure 2B illustrates the improvement resulting from the addition of joint angular position and velocity inputs (black curve). It is hoped that incorporation of limb state information into the prediction of kinetic signals may lead to improved on-line BMI performance.

Prediction of limb muscle activity

Miller's group has begun to examine the potential for BMI technology to be applied directly to the paralyzed muscles of a patient's limb, through the prediction of muscle activity (EMG). EMG signals are stochastic and noisy by nature, yet decoders like those described above accounted for as much as 70–75% of the EMG variance. Figure 2C shows representative examples during three successive reaches. The accuracy of the predictions made by specific decoders remained statistically unchanged during a typical experimental session and, more importantly, decreased by only 15–20% over periods as long as 2 weeks. Similar EMG predictions have been made by Nicolelis' group (Carmena et al., 2003).

The Freehand neuroprosthesis electrically stimulates muscles of the arm and hand to restore grasp function in human patients with spinal cord injury (Keith et al., 1989; Peckham et al., 2001). However, because of the limited voluntary control options currently available to these patients, the stimulus patterns must be preprogrammed. Miller's group is now using real-time EMG predictions to control a four-channel stimulator and intramuscular electrodes in a monkey (Fig. 1C). The system is designed to provide voluntary control of wrist force to a monkey whose forearm flexor musculature has been paralyzed by a peripheral nerve block. In the blocked state, the system allowed the monkey to generate forces well above what he could achieve without stimulation. It is important to note that the monkey was able to make use of the system almost immediately, without the need for retraining. The results suggest that cortically generated EMG predictions, combined with a system like Freehand, may allow patients to achieve a broader range of more dexterous hand use than is currently possible.

Visual decoding

Unlike the experiments described above, the lack of arm movement in paralyzed patients makes it impossible to directly calculate the transfer function between motor cortical activity and desired output. Others have shown that real-time decoding parameters can be computed from neural recordings made while monkeys passively observe the corresponding movements (Wahnoun et al., 2006) or when paralyzed patients imagine tracking a moving cursor (Hochberg et al., 2006). Hatsopoulos and his colleagues have compared the performance of such observation-based decoders with that of movement-based decoders in providing real-time control. Although the approach works, they discovered that performance was superior when training and testing conditions were congruent. That is, if the decoder was trained during passive observation, brain control was better when there was no arm movement. Likewise, if the decoder was trained during movement, brain control was superior when the arm was moving. These differences might be related to the presence or absence of somatosensory feedback from the limb.

Evidence of the nonlinear encoding of movement

Although the manner in which single neurons encode motor-related information has been studied extensively, the quest for a theoretical understanding of the information encoded within even small ensembles of neurons poses daunting challenges. If one considers the state of the brain to be determined by the discharge of its constituent neurons, the potential dimensionality of the state space is enormous: in humans, corticospinal fibers alone number ∼1,000,000. However, the activity of all these neurons is not mutually independent. Correlations within a set of neurons means that the information from some neurons can be ignored, without an overall loss of information. Principle components analysis (PCA) is a standard method by which data points from such a system can be projected onto a lower dimensional structure. The operation is analogous to the shadow of a three-dimensional object being projected by a beam of light onto a two-dimensional screen. By varying the orientation of the object, one can change the degree to which the information contained by the object's shape appears in the two-dimensional projection. PCA essentially offers a way to find the orientation that preserves the most information, whether the projection is from three to two dimensions, or from 10⁶ to 10¹.

However, PCA is a linear operation. The low-dimensional embedding of activity within the brain could in principle be either linear or nonlinear. In either case, it might offer a compact, task-related description of ensemble neural activity during movement (Georgopoulos et al., 1983; Seung, 1996; Zhang, 1996). The groups of Solla and Miller collaborated to analyze data collected from M1 during a multi-target reaching task. They discovered that a particular nonlinear dimensionality reduction technique (Isomap) was considerably more successful than PCA in identifying low-dimensional structures within the data. Isomap allows the same rotational operations described above for PCA, but, in addition, the high-dimensional object can be stretched arbitrarily along any combinations of axes. This nonlinear operation preserved as much as two to three times more information about the monkey's movements than did PCA, and, when plotted in two dimensions, the projections clearly reflected the geometry of the task.

The results shed additional light on hypotheses concerning the rate coding of M1 neurons. Most, if not all, of these hypotheses, even those that attempt to reconcile “kinetic versus kinematic” views of M1 activity (Todorov, 2000), build on the notion of cells whose discharge is described as cosine tuning around a preferred direction, be it in force, velocity, or displacement coordinates. Cosine tuning implies linear coding and would lead to population activities whose distribution in state space should be well captured by PCA. We find that the heterogeneous tuning properties of M1 cells, taken as a whole, generate a distribution that is fundamentally “curved” and requires nonlinear methods to capture its structure. The potential application to BMI control remains to be explored.

The afferent interface: the potential role of somatosensation

In all current BMI applications, feedback to the subject has been supplied entirely through the intact visual or auditory systems. However, human patients lacking proprioception (the innate sense of limb position distinct from touch) have greatly impoverished movement, even if their motor capacity itself is undiminished (Sanes et al., 1985; Sainburg et al., 1993; Abbott, 2006). Vision of the limbs diminishes these deficits but by no means eliminates them. Patients appear to have difficulty forming accurate internal models of limb dynamics (Gordon et al., 1995), and corrections tend to come late and often cause new errors, resulting in jerky, unstable movements. It must be assumed that the lack of proprioceptive feedback in BMIs also contributes to their characteristically slow, unstable movements.

Natural somatosensation

The Hatsopoulos laboratory is trying to augment BMI control with natural proprioception. The method involves guiding the animal's own arm, using cortically derived position command signals sent to the exoskeleton as well as to the cursor (Fig. 1A,B). The torque motors of the exoskeleton thus control limb position, ideally causing the monkey's limb to act as afferent source of feedback, rather than an efferent source of movement. However, initial attempts to compare BMI performance with and without guidance from the exoskeleton were confounded by the fact that the animal actively resisted the movements. The group is planning to conduct experiments in which the arm is temporarily paralyzed by Botulinum toxin (Botox) injections to the major muscles of the arm.

Electrical stimulation of the somatosensory cortex

Making use of the animal's natural somatosensory system may help to clarify the interacting roles of vision and proprioception, but it would have no direct clinical relevance to most patients. One potentially useful option is artificial somatosensation supplied by electrical stimulation of the CNS, similar in concept to that of the clinically successful cochlear implant (Loeb, 1990; Wilson et al., 2003).

Relatively little is known of the physiology of area 3a, the proprioceptive region of the somatosensory cortex (S1), compared with its tactile portion, 3b. In part, this may be because of its location deep in a sulcus of the brain. In preliminary experiments, Miller's group has shown that a monkey can detect and discriminate between electrical stimuli of different frequencies in 3a (London et al., 2008). Miller's experiments were modeled on the much more extensive electrical stimulation studies of area 3b by Romo and colleagues (Romo et al., 1998, 2000). In those experiments, monkeys learned to discriminate different stimulus frequencies, whether the stimuli were applied mechanically to the finger or electrically through a microelectrode to the brain. The reaction times and the shape of the psychophysical frequency discrimination curves were essentially the same for electrical and mechanical stimuli, suggesting that the approach may ultimately be useful to convey complex, time-varying information about limb state.

Development of the somatosensory percept

In such a discrimination task, repeated, identical presentations of a near-threshold stimulus may lead to different behavioral responses. The emergence of the different perceptions must be related to variations in neural activity that is no longer locked rigidly to the physical stimulus. The question then is how the stimulus-locked representations that first appear in S1 ultimately give rise to more variable sensory perception (Lamme and Roelfsema, 2000).

Remarkably, the psychophysical threshold for stimulus detection matches quite closely the sensitivity of single, primary sensory neurons in the parietal cortex (Mountcastle et al., 1969; Newsome et al., 1989). Conversely, activity recorded from neurons in the frontal lobe has been shown to correlate more closely with the animal's perception of stimulus characteristics (Thompson and Schall, 1999; Gold and Shadlen, 2000; Romo et al., 2004). Lafuente and Romo investigated this transformation by training monkeys to indicate the presence or absence of a small vibratory stimulus while recording the activity of single neurons in various cortical areas potentially involved in processing somatosensory information. Activity patterns of neurons in primary somatosensory areas (3b, 1, 2) and area 5, exquisitely reflected the physical properties of the vibratory stimuli but gave no information as to how the monkeys perceived the stimulus. In contrast, activity of neurons in the ventral and dorsal premotor cortices and the supplementary motor area more closely covaried with the monkeys' behavioral reports, even when those reports did not correctly reflect the stimulus characteristics. The results are consistent with perception being a phenomenon that builds through time and through propagation across multiple cortical areas (Fig. 3).

Figure 3.

Flow of somatic information from the sensory cortices of the parietal lobe to the premotor and motor cortices of the frontal lobe. Color scale indicates the latency to activation of each cortical area after a 20 Hz vibratory stimulus applied to a fingertip. 1, 2, 3b, 5, Brodmann areas 1, 2, 3b, and 5; S2, secondary somatosensory cortex; PMd, PMv, dorsal and ventral premotor cortices; SMA, supplementary motor area.

Injury-related changes in tactile representation within S1

If stimulation of S1 were to be used to supply artificial somatosensation, an important consideration is that the approach would ultimately need to be implemented in patients whose nervous system has suffered serious damage (Hochberg et al., 2006). After spinal cord injury, neurons in the affected area of the somatosensory cortex do not respond well to passive cutaneous stimulation, and recovery is quite limited (Jain et al., 1995, 2003). Recent studies in the Moxon laboratory examined how sensorimotor information is encoded in normal animals and how this encoding is affected by spinal cord injury. Before hemisection, cells within a single cortical column encode passive sensory information using large, heterogeneous receptive fields covering the digits and palm (Tutunculer et al., 2006), and the timing between spikes contributes to a distributed spatiotemporal code (Foffani and Moxon, 2004; Foffani et al., 2004). In the awake animal, recorded neuronal activity can be used to encode the placement of footfalls on a treadmill with a sensitivity of >90%.

As expected, after spinal cord injury, these cells suffered a significant loss in responsiveness to passive sensory stimulation. However, during locomotion, these same neurons actually responded earlier and more strongly to footfalls than before the hemisection. Footfall detection was no different before and after the lesion. These results suggest that the loss of passive sensory input attributable to spinal cord injury is compensated by active processing during locomotion. The cortex seems to combine expectation of the coming event with the passive sensory input, possibly integrating information from other senses as well. Therefore, in the context of an afferent BMI, it is likely that the somatosensory cortex will remain capable of detecting electrical stimulation supplying limb state information and relaying the sensation to the higher cortical areas that form perception.

Summary

In this mini-symposium, we will review several of the efferent BMI applications by which limb movements might be controlled by kinetic command signals. Given the heterogeneity of signals that have been found in M1, it is an open question whether such approaches may prove more accurate or natural than existing ones have been. Existing BMI applications lack the somatosensory feedback that is critical for normal movement. We will review the propagation of signals related to tactile perception through different cortical areas and over time, as well as the changes in cortical responses to peripheral stimulation after spinal cord injury. We will explore the feasibility of incorporating somatosensory feedback into an afferent BMI application, in a manner that would complement the efferent interface.