The science and engineering behind sensitized brain-controlled bionic hands

Abstract

Advances in our understanding of brain function, along with the development of neural interfaces that allow for the monitoring and activation of neurons, have paved the way for brain-machine interfaces (BMIs), which harness neural signals to reanimate the limbs via electrical activation of the muscles or to control extracorporeal devices, thereby bypassing the muscles and senses altogether. BMIs consist of reading out motor intent from the neuronal responses monitored in motor regions of the brain and executing intended movements with bionic limbs, reanimated limbs, or exoskeletons. BMIs also allow for the restoration of the sense of touch by electrically activating neurons in somatosensory regions of the brain, thereby evoking vivid tactile sensations and conveying feedback about object interactions. In this review, we discuss the neural mechanisms of motor control and somatosensation in able-bodied individuals and describe approaches to use neuronal responses as control signals for movement restoration and to activate residual sensory pathways to restore touch. Although the focus of the review is on intracortical approaches, we also describe alternative signal sources for control and noninvasive strategies for sensory restoration.

CLINICAL HIGHLIGHTS

1. Brain-machine interfaces (BMIs) promise to restore independence to individuals with paralysis.

2. BMIs harness signals from brain regions associated with motor control in the frontal and/or parietal lobes to control robotic arms or exoskeletons or to reanimate the native limb via electrical stimulation of the muscles. Our evolving understanding of brain activity in motor areas, and its relation with movement intention, is providing new avenues to increase BMI performance. A variety of mathematical approaches are deployed to infer movement intent from patterns of neuronal activity.

3. BMIs deliver electrical stimulation to brain regions associated with somatosensation to convey information about the state of the bionic limb and about its interactions with objects. A guiding principle of artificial somatosensory feedback is biomimicry, which consists of natural patterns of neural activation to the extent possible.

4. BMIs based on invasive neural interfaces–electrodes that penetrate the cortex–outperform BMIs based on less invasive interfaces–extracranial or even electrocorticographic.

5. The development of BMIs raises significant ethical issues that will need to be addressed as deployment nears.

1. INTRODUCTION

Over the last century, our understanding of the neural basis of behavior has undergone an astounding expansion, from a vague sense that the brain gives rise to the mind to a sophisticated integration of biological insight and mathematical modeling. In recent years, advances in neuroscience, coupled with breakthroughs in engineering, have been leveraged to develop systems that allow the nervous system to interact directly with extracorporeal machines, bypassing the muscles and the senses (FIGURE 1). These brain-machine interfaces (BMIs) have the potential to restore lost sensorimotor function, for example in tetraplegia, typically caused by spinal cord injury (SCI) or neurodegeneration, which results in a severe loss of independence.

FIGURE 1.

Brain-machine interfaces for restoring reaching and grasping. Intended movements are decoded in real time from neuronal signals monitored in motor regions of the brain and translated into electrical activation of muscles or movements of a prosthetic limb. The output of sensors on the bionic hand, which convey information about object interactions, is converted in real time into electrical pulse trains delivered to the somatosensory regions of the brain.

One strategy to restore independence to individuals with tetraplegia is to harness signals from their brain to control robotic limbs or exoskeletons or to reanimate muscles. Indeed, in tetraplegia, the brain regions involved in motor control are typically intact and thus still carry information about intended movements even though they no longer activate muscles. Relatively simple mappings can be developed to infer motor intent, in real time, from continuously monitored neural activity. Decoded movements can then be executed by a robotic limb or exoskeleton or by electrically activating the muscles, which are also intact though no longer connected to the central nervous system.

To achieve dexterous control of a bionic hand requires not only the restoration of volitional movement but also the restoration of somatosensory feedback. Indeed, our ability to interact with our environment relies critically on somatosensory signals—tactile and proprioceptive—that carry information about the state of our limbs and about their interactions with objects (1–3). This sensory feedback can be restored by converting the output of sensors on the bionic hand into patterns of electrical stimulation of somatosensory regions of the brain, thereby giving rise to vivid somatosensory percepts.

Over the last few years, BMIs have achieved remarkable scientific and technological feats. One paralyzed subject was able to write words fluidly by controlling a computer with brain signals (4). Another was able to reach and grasp with his own limb (5). Yet another not only could control an anthropomorphic bionic arm by thought alone but could actually feel objects with it (6)!

In the present review, we describe the neural mechanisms that mediate movement generation in able-bodied individuals and discuss how signals from the central nervous system can be monitored to infer intended movement and used to drive bionic or reanimated limbs. We then summarize what is known about the neural basis of somatosensation—touch and proprioception—in the brain and consider how this knowledge can be leveraged to restore these critical sensory modalities via electrical activation of somatosensory neurons. Throughout, we attempt to identify the challenges that remain for achieving a dexterous bionic limb and propose ways in which these challenges might be overcome.

2. ETIOLOGIES OF MOTOR IMPAIRMENT AND RELEVANT ASSISTIVE TECHNOLOGIES

Tetraplegia—sometimes called quadriplegia (though this word awkwardly comprises a Latin and a Greek root)—refers to the partial or complete loss of the use of all four limbs. Individuals with tetraplegia are often dependent on caregivers, which results in a severe reduction in quality of life and great financial cost. Tetraplegia can be caused by a variety of conditions including spinal cord injury (SCI), neuromuscular disorders such as amyotrophic lateral sclerosis (ALS) and cerebral palsy, stroke, and traumatic brain injury. In people with SCI, the degree of impairment depends on the location and extent of the lesion along the spinal cord. Higher lesions result in more generalized paralysis and desensitization (FIGURE 2A), and some lesions are only partial whereas others are complete. After an acute recovery phase, people with chronic SCI can retain a stable degree of motor function that lasts decades (7). In contrast, people with ALS undergo a gradual loss of motor neurons in the brain and spinal cord, resulting in a progressive deterioration of motor function that ultimately affects limb function, chewing, swallowing, and breathing (8). In contrast to SCI and ALS, where the mechanisms underlying volitional movement in the brain may be largely undisturbed, traumatic brain injuries and stroke are heterogeneous in nature and span the full range of potential motor and cognitive impairment, with the nature and degree of impairment, and potential for recovery, highly dependent on the specifics of the injury such as the affected brain regions. In the case of stroke, implanted BMIs have been developed for people whose injuries largely affect motor output, such as those with pontine strokes (9, 10). A separate field of rehabilitative BMIs seeks to use neuroplasticity and motor learning to enhance motor recovery and has primarily focused on leveraging noninvasive interfaces such as electroencephalography (EEG) (11, 12).

FIGURE 2.

Organization of the spinal cord and implications for brain-machine interfaces. A: diagram depicting spinal nerves and the motor and sensory functions they mediate (reproduced from spinal-research.org). Arm and hand function can be severely impacted by injuries to cervical spinal regions, and the amount of function retained is largely dependent on the level of injury. B: assistive priorities assessed via surveys of people with spinal cord injury. Interest in different control modalities and levels of performance varies depending on the level of injury. wpm, Words per minute. Adapted from Ref. 14, with permission from Journal of Neural Engineering.

People with tetraplegia consistently identify arm and hand function as among their highest priorities for functional restoration (13–16), although priorities depend on the level of injury (FIGURE 2B; Ref. 14). In surveys of people with SCI, those with high tetraplegia (C₁–C₄ injuries), who have the most limited arm and hand function and sensation, are eager to adopt a wide range of BMIs, even if performance is somewhat limited. In contrast, people with low tetraplegia (C₅–C₇), who retain some arm and wrist function, expect greater dexterity. Functional aspirations include restoration of high-performance typing and computer control and/or restoration of reaching and grasping with fast robotic arms or, ideally, with their native limb. Although the invasiveness of neural interfaces influences their appeal, practical factors such as increased mobility and cosmetic factors such as reduced device visibility are equally important. For example, noninvasive systems based on EEG, with electrodes either integrated into a headcap or glued to the scalp, are no more appealing than wireless implanted systems. Wireless interfaces are of particular interest: respondents with SCI were twice as likely to be interested in implanted BMIs if they could be made wireless (14).

Robotic arms, actuated exoskeletons, and functional electrical stimulation (FES) are potential avenues to restore reaching and grasping using BMIs (5, 9, 17–24). Robotic arms used in assistive studies can vary widely: some arms are designed for use by amputees, while others are designed to mount on wheelchairs or even for industrial applications. Exoskeletons that drive both arm and hand movements are in comparatively early stages. Critical design considerations for assistive robots include the weight, power consumption, performance/speed, compliance/safety, reliability, and the degree to which they are anthropomorphic (look and move the way native human arms do).

FES involves applying patterned electrical stimulation to the muscles and/or nerves to reanimate the paralyzed limb (25). Electrical stimulation can be delivered to the skin surface, to the muscle directly, or to the nerves that innervate the muscles. FES has been successfully deployed as a rehabilitation tool (25), for example to help “exercise” paralyzed muscles with the goal of maintaining muscle mass, or to assist or regulate residual muscle function in individuals with mid- to low-level cervical SCI or upper extremity paralysis due to stroke (26–31). Skin surface stimulation has provided valuable proofs of concept for intracortical BMI-FES systems (18). For long-term clinical translation, intramuscular electrodes (5) provide more selective stimulation, are highly durable [<1% of electrodes fail over 3 yr (30)], and are anchored in place, allowing seamless day-to-day use.

BMIs for reach and grasp (5, 6, 9, 17–24) provide a richer and more natural set of control options than traditional assistive devices, which rely primarily on movements of spared body regions (e.g., by monitoring muscle activity in the neck or face, tracking eye or head posture, or using the mouth to operate a manipulandum). Traditional devices involve unnatural control strategies, ranging from somewhat straightforward, e.g., mapping head movements onto the movement of a computer cursor or robotic arm, to more complex and counterintuitive, e.g., common “sip-and-puff” devices that translate air pressure from sucking or blowing on a straw into wheelchair movements. Another disadvantage of traditional assistive devices is that they provide control in a serial fashion so that the user can only do one thing at a time (32). In contrast, BMIs have the potential for simultaneous control of many degrees of freedom, which more closely approximates natural motor function.

In addition to the restoration of limb function, control of communication devices and computers is also a priority for people with tetraplegia. Individuals with ALS or high-level SCI might communicate primarily through alternative and augmentative communication devices, which typically monitor retained movement abilities such as eye or head movements, and may require caregiver assistance (33). Furthermore, although computers, tablets, and phones are increasingly integral to daily life, control is typically manual and thus ill suited for paralyzed individuals. BMIs can provide direct interfaces with computers to restore rapid typing and communication and more general usage, including web browsing, e-mail, chatting, and playing music (4, 34–38).

3. EFFERENT SIGNALS FOR PROSTHETIC CONTROL

Since the dawn of humanity, our interactions with the environment have relied on muscular output, whether it be for locomotion, object interactions, or communication. The revolutionary idea underlying BMIs is that signals from the brain can be harnessed to control extracorporeal devices directly, thereby bypassing the muscles (39). This strategy requires a mapping between patterns of neural activity and the behavior of the device. For bionic arms, this mapping can be informed by a basic understanding of how the brain controls the limbs in able-bodied individuals.

Most theories of motor control posit that movement planning involves a series of processing steps to transform a movement goal (such as a reach to a certain location) into a pattern of muscle activity that executes the movement and completes the action (40). This process is distributed across multiple brain regions with distinct anatomical and functional properties. Intracortical BMIs have largely focused on leveraging control signals from two brain regions: the motor cortex and the posterior parietal cortex. These areas are thought to be situated at distinct levels in the motor control hierarchy for reaching and grasping: motor cortex is implicated in low-level aspects of movement planning and execution and more closely tied to muscle activity (40, 41), and posterior parietal cortex (PPC) is involved in higher-level cognitive functions related to sensory-motor integration and intention formation (42).

The anatomy and physiology of brain areas associated with motor control have been widely studied in monkeys, and these areas are thought to have homologs in humans. However, there remains a surprising lack of consensus on the basic identity of some these human homologs (43), as discussed below. Nonetheless, studies with monkeys have served as a critical guide, both for understanding the basic anatomy and physiology of the motor control system and for testing new approaches to best harness control signals from the brain.

3.1. Motor Cortex

In the late nineteenth century, electrical stimulation of portions of the mammalian cortical surface was found to evoke movements on the contralateral side of the body (44), a finding that implicated for the first time a specific region of the cerebral cortex in the control of movement. Since then, motor cortex has been recognized as a critical hub for movement generation. In monkeys, motor cortex stretches over an expansive area of the frontal lobe and is traditionally divided into seven well-defined cortical fields (FIGURE 3A; reviewed in Refs. 45, 46). The primary motor cortex (M1, Brodmann’s area 4), which occupies the precentral gyrus and the rostral bank of the central sulcus, is the output stage of critical networks for both reaching and grasping that include premotor and parietal motor areas. Accordingly, injuries to M1 or to its downstream targets result in profound deficits in movement control. For example, in classic studies with monkeys, lesions of the output from M1 to the spinal cord resulted in permanent deficits in reaching and grasping and in the abolition of independent digit movements (47, 48). Anterior to M1 is Brodmann’s area 6, which consists of several subdivisions considered to be situated higher in the motor control hierarchy than M1: the dorsal and ventral premotor cortices, which each have rostral and caudal divisions (PMdr, PMdc, PMvr, PMvc), the supplementary motor area (SMA), and the pre-supplementary motor area (pre-SMA); these areas are often referred to by field numbers (F1–F7, respectively) (46, 49).

FIGURE 3.

Motor cortical areas contain command signals for intracortical neural prostheses. A: diagram showing a lateral view of the monkey cerebral cortex. Motor cortex is a region of the frontal lobe that spans Brodmann areas 4 and 6. Area 4 contains primary motor cortex (M1), whereas area 6 is subdivided into 6 regions: dorsal and ventral premotor cortices, which each have rostral and caudal divisions (PMdr, PMdc, PMvr, PMvc), supplementary motor area (SMA), and pre-supplementary motor area (pre-SMA); the latter regions extend into cortical areas along the medial wall. In the parietal lobe, important regions for reach and grasp are located within or adjacent to the intraparietal (IP) sulcus. B: sagittal schematic of the monkey precentral gyrus (PcG) and anterior bank of the central sulcus (CS), showing area 6, area 4 (M1), and area 3a (primary somatosensory cortex). Cortico-motor neuronal (CM) projections primarily originate from sulcal M1 and show coarse somatotopic organization (adapted from Ref. 57, with permission from Proceedings of the National Academy of Sciences of the United States of America). C: subdivisions of human cortex as illustrated by Vogt and Vogt, which designated the majority of the precentral gyrus as Brodmann’s area 6. F.R., fissure of Rolando (central sulcus). Image inspired from Ref. 75. D: preoperative functional MRI activation maps during attempted movements by a paralyzed clinical trial participant. Approximate locations of 2 Utah arrays implanted in the hand area of the precentral gyrus are shown as black squares in the inset. Adapted from Ref. 20, with permission from Lancet. E: the motor homunculus depicts the broad somatotopic arrangement of motor cortex as estimated by movements evoked by electrical stimulation of the cortical surface. Although useful to illustrate the general organization of leg, arm, and face regions and the gradients within, the homunculus is not meant to be a precise parcellation of cortex into distinct regions. Adapted from Ref. 78, with permission from Journal of Neuroscience.

As mentioned above, motor cortex is organized hierarchically: M1 is generally associated with movement execution and contains the most direct connections to arm and hand muscles, whereas PMd and PMv are associated with higher-level movement planning. SMA and pre-SMA are associated with many higher-order aspects of motor control, including bimanual coordination, movement timing, and action sequencing (50–54). Although premotor cortical regions are commonly considered to lie within Brodmann’s area 6, anatomical evidence suggests that multiple portions of the cingulate cortex, situated within the medial wall and buried under surface cortical structures, might also be considered premotor areas, in that they make direct projections to M1 (55, 56). To date, the neural activity within these areas has been poorly characterized because they are more difficult to access.

All movements of the arm and hand are ultimately driven by motor neurons in the spinal cord, whose axons innervate the muscles themselves. The cortical control of movement is thus mediated by pathways that directly or indirectly drive the spinal cord. The corticospinal tract (CST) directly innervates neurons in the spinal cord and comprises projections from multiple cortical regions, including primary motor and premotor cortices, as well as the somatosensory, parietal, and cingulate cortices. In most mammals, axons in the CST synapse onto spinal interneurons, which play a role in both reflexive movements and pattern generation. In several primates—including monkeys, apes, and humans—an additional pathway provides more direct cortical control of the arm and hand (57). In these species, a fraction of the CST consists of cortico-motoneuronal (CM) projections, which synapse onto spinal motoneurons that directly innervate muscles. CM neurons thus provide a more direct line from cortex to muscles than do other cortical neurons. CM projections are particularly prominent for muscles that drive manual behaviors, and most CM-projecting neurons in cortex are located in the anterior bank of the central sulcus (FIGURE 3B; Ref. 57). Accordingly, M1 in monkeys is frequently subdivided into caudal (“new”) and rostral (“old”) regions (57–60), distinguishable based on the presence or absence of CM-projecting neurons (57). Evidence for a similar division exists in humans as well (61).

Human intracortical BMIs typically target the hand representation in the precentral gyrus (FIGURE 3, C and D), which can be localized based on a distinct knob-shaped anatomical landmark (22, 62). Conflicting terminology is used when referring to the targeted region on the precentral gyrus. Sometimes the generic term “motor cortex” (5, 20, 38) is used, while in other cases it is referred to as “primary motor cortex” or “M1” (9, 18, 22, 63, 64). A recent review of the anatomical evidence (43) concludes that, in fact, most of the human precentral gyrus is actually occupied by the caudal portion of Brodmann’s area 6 (61, 65–69), a homolog of PMd in macaques (70–72). Some of the confusion may arise from discrepancies in classic illustrations: although Brodmann’s original illustrations depicted his area 4 as occupying much of the gyral surface (73), Vogt and Vogt depict the precentral gyrus as being dominated by Brodmann’s area 6 (FIGURE 3C; Refs. 74, 75). Arm and hand areas of Brodmann’s area 4 are largely confined to the anterior bank of the central sulcus (61, 66–69). These areas would be difficult or impossible to access with intracortical electrode arrays that are currently approved for human use, which penetrate 1–1.5 mm into cortex. Recent evidence suggests that the only portions of area 4 that emerge from the central sulcus are medial regions associated with leg movements (65, 67–69).

Somatotopic organization in motor cortex has been recognized for more than a century (76). The most well-known illustration of motor cortical somatotopy is Penfield’s homunculus (FIGURE 3E; Refs. 77, 78), a classic diagram depicting movements elicited by stimulating different locations of the motor cortical surface during intraoperative studies with epilepsy patients. At the medial extent of the precentral gyrus surface stimulation evoked movements of the leg, and as stimulation proceeded more laterally evoked movements transitioned to proximal and then distal arm and hand movements, and then neck, face, jaw, and tongue movements. Importantly, Penfield and Rasmussen considered this homunculus a useful memory aid and not a precise parcellation of cortex into areas responsible for movements of specific joints (77). A review of decades of literature that included studies with more precise stimulation methods [intracortical microstimulation (ICMS)] concluded that stimulation results are indeed consistent with a coarse organization of motor cortex into leg, arm, and face regions but not with discrete, well-ordered subdivisions of different within-limb movements (76).

3.2. Posterior Parietal Cortex

In contrast to the frontal lobe, whose role in controlling movements has been studied since the nineteenth century, posterior areas of the parietal lobe have been implicated in movement control only relatively recently. Posterior parietal cortex (PPC) typically refers to the region of the parietal lobe that lies posterior to the postcentral gyrus (i.e., posterior to somatosensory cortex). Areas of PPC receive diverse inputs from various regions of sensory cortex. Classically, PPC was considered an “association area” and hypothesized to carry higher-order sensory representations that integrated information across sensory modalities and were well suited to guide action selection (42, 79, 80). Closer examination revealed that the responses of neurons in PPC were not solely tied to sensory stimuli but were also dependent on motor intent (81). Two regions of PPC seem to be specifically implicated in movements of the upper limbs, namely the anterior intraparietal area (AIP) and the parietal reach region (PRR) (42, 82). AIP is strongly and reciprocally connected with PMdr (F5) in motor cortex, which, together with M1, forms a network critical for grasping and manipulating objects (42, 46, 79, 83). Neurons in AIP often selectively encode specific grasping movements (84). Neurons in PRR often encode reaching targets (42). Recently, putative homologs of AIP and Brodmann’s area 5—which in monkeys is adjacent to PRR and appears to also encode reach plans (85)—have been identified in humans (17, 86, 87).

4. NEURAL CORRELATES OF MOVEMENT INTENTION

The most common approach to BMI control via an intracortical interface is to obtain neural signals about low-level aspects of end effector movements—typically from motor cortex—and execute the intended movements with a bionic hand or through electrical activation of the muscles. Yet, despite decades of research, the precise relationship between motor cortical activity and movements is still unclear. Understanding the factors that shape motor cortical activity and its relationship to behavior is a critical step in harnessing it to achieve dexterous BMIs.

4.1. Representations of Movement Variables

A fundamental concept underlying the interpretation of motor cortical neuronal activity is that neuronal responses encode specific externally measurable movement parameters, e.g., joint angle or hand velocity. This representational view parallels a pervasive view in sensory neuroscience, inspired by the oft-repeated observation that the responses of sensory neurons are systematically modulated by changing specific stimulus features, e.g., the orientation of an edge or the frequency of an acoustic tone. Sensory neurons are then described as representing or encoding these stimulus features. Similarly, the responses of neurons in motor cortex typically covary with external movement parameters, and motor cortical neurons are frequently described as representing or encoding specific aspects of movement. In a landmark study (88), Georgopoulos and colleagues recorded activity from neurons in monkey M1 and quantified the relation between their firing rates and end point reach angles in a planar reaching task (FIGURE 4A). They found that the responses of individual neurons were strongly modulated by the direction in which the monkey reached. Neurons tended to respond maximally when the animal reached in a particular direction—its so-called preferred direction—and less so when it reached in other directions. In fact, the firing rate of each neuron could be approximated by the (scaled) cosine of the angular difference between the reach direction and the neuron’s preferred direction. Within this framework, each neuron can be described simply by its preferred direction. In the decades since, motor cortical neurons have been found to covary with many other movement parameters, and the list now includes many kinematic variables (e.g., position, velocity, speed, and acceleration of the hand, joint angles and velocities, etc.) and kinetic variables (end point forces or torques exerted on the joints) (40). For BMI applications, the representational model lends itself to simple strategies for decoding movement intention, as the dependence between neural firing rates and external movement parameters can be inverted to decode these parameters from firing rates (detailed in sect. 5.2).

FIGURE 4.

Frameworks for interpreting motor cortical activity. A: the firing rates of an individual neuron in primary motor cortex (M1) are highly dependent on the reach direction. This neuron’s preferred direction is ∼180°. Adapted from Ref. 88, with permission from Journal of Neuroscience. B: in a simple 3-neuron (n) example, the population’s activity at each time step (t) can be represented as a point in a 3-dimensional (3-D) state space, where the firing rate of each individual neuron represents an axis. Such analyses often reveal that the activity of many neurons can be captured by a modest number of abstract, time-varying latent factors. In this example, 3-D population activity is captured by a lower-dimensional manifold (a 2-D plane), whose axes are defined by the latent factor dimensions x₁ and x₂. Adapted from Ref. 106, with permission from Nature Neuroscience. C: in a state space view, neural population activity at the transition from preparation to execution exhibits consistent dynamics. The neural state achieved during the delay period (green-red dots) predicts the subsequent trajectory of movement activity (green-red lines). Each dot/line is a single reach condition (inset). Adapted from Ref. 98, with permission from Nature. D: robust dynamical systems must avoid regions of high tangling, in which slight perturbations to the system’s state can result in different trajectories. E: in this conceptual example, the target pattern of activity (output dimensions 1 and 2) exhibits high tangling. A dynamical system can use additional internal dimensions that detangle the system’s state, which are not observable at the output level. Adapted from Ref. 117, with permission from Neuron.

4.2. Preparatory Activity

Another hallmark of many neurons in both primary motor and premotor cortices is that they are active well before the onset of movements (89, 90). In monkeys, such activity is frequently studied with instructed delay paradigms, in which information about the upcoming movement is provided to the subject before the movement is executed. During this delay period, single neurons often exhibit distinct modulation that depends on the specifics of the upcoming movement. Such preparatory activity is predictive of both reaction times and movement variability (91–96), and disrupting preparatory activity delays the onset of movement (94). Traditionally, representational models of neuronal tuning viewed preparatory activity as a “subthreshold” version of the activity observed during movement execution (89). For example, if a given neuron encodes rightward movements, then it may be beneficial for movement preparation to activate that neuron before moving to the right, but at a level that does not trigger overt movement. However, the activity of individual neurons during preparation is only weakly predictive of their activity during execution (97). In contrast, at the level of populations of neurons, preparatory activity shows clear structure that has a consistent relationship with activity during movement execution (41, 97, 98). The revelation of a close link between preparation and execution at the population level is one of the key successes of the dynamical systems view outlined below (see sect. 4.3). Posterior parietal cortex also exhibits prominent preparatory activity (82, 83, 100, 101).

4.3. Manifolds and Dynamics

The representational model has been successful in demonstrating correlations between neural activity in motor cortex and many movement parameters. However, as the list of movement parameters has grown longer, it has become apparent that correlations with movement parameters can often be spurious (102) and that representational models fail to explain many aspects of neuronal responses (103, 104).

In parallel, the widespread adoption of techniques to simultaneously record from many neurons (105) has enabled a shift in focus from studying the relation between individual neurons’ activity and externally measurable parameters to investigating the structure of the neuronal activity itself. One striking finding across many brain areas is that much of the observed activity of large neural populations can be described as a combination of a modest number of “latent factors” (106–108)—abstract, time-varying patterns that reflect the coordination of responses across neurons (FIGURE 4B). This makes intuitive sense: as the recorded neurons participate in a common underlying network, their responses cannot be completely independent. A focus on latent variables shifts emphasis from understanding the responses of individual neurons to understanding patterns that are distributed across the network.

The emerging dynamical systems view focuses on understanding how computations are performed by the collective activity of populations of motor cortical neurons. This perspective treats motor cortex as a computational engine that translates high-level movement intention into the complex, time-varying patterns of muscle activity required to execute a movement (41, 108–112). A key principle underlying dynamical systems is that the system’s future state is largely predictable based on its current state and inputs (detailed in FIGURE 5). In the case of a neural system, the system’s state is the current pattern of neuronal activity across the population, and inputs represent the activity of unmeasured neurons (e.g., from other brain areas), including those that relay sensory information from the body and environment.

FIGURE 5.

Conceptual dynamical system: a frictionless 1-dimensional (1-D) pendulum. A and B: a pendulum released from different starting positions will trace out different positions (p) and velocities (v) over time. C: when viewed in a 2-dimensional state space, the state of the pendulum x follows simple dynamical rules (i.e., the pendulum’s equations of motion) that are invariant to the point of release. D–F: if the pendulum’s motion is perturbed by an external factor, the evolution of the system’s state no longer follows autonomous dynamic rules. Adapted from Ref. 108, with permission from Journal of Neuroscience.

In recent years, a careful examination of motor cortical activity has borne out many predictions of the dynamical systems view (41, 108). One key prediction is that during a movement the initial conditions of the system (as might be observed during movement preparation) should largely predict subsequent activity. For example, in an instructed delay paradigm, the progression in motor cortical state during movement execution is largely predictable based on the state of motor cortex during the delay period (FIGURE 4C; Refs. 97, 98, 113). Across a variety of movements, the same dynamical rules govern how neural activity evolves over time.

Another prediction of the dynamical systems view is that not all of the activity in motor cortex should relate to movement variables. Rather, a substantial fraction of the neural activity may relate to internal processes needed to accomplish computational demands. For example, movement preparation and execution entail substantially different neuronal dynamics (114, 115). The transition between these dynamics manifests as a large translation in state space, hypothesized to trigger movement generation dynamics (116). This translation is highly prominent (often explaining more than half the variance in the neural population activity) and independent of the specific movement. Another place where internal processes play a critical role is in the process of pattern generation. A robust dynamical system should avoid “tangling,” i.e., situations in which similar states lead to divergent future states (FIGURE 4D; Ref. 117). If dynamics were organized such that two nearby points led to divergent trajectories in state space, then a small amount of noise in the system might lead to dramatic differences in the evolution of the neural state and, consequently, in the evoked movement. Thus, for a dynamical system to reliably generate patterns of output, e.g., patterns of muscle activity for distinct movements, the system must have additional, internal dimensions that are not directly linked to the output but instead serve to “pull apart” the system’s state to avoid tangling (FIGURE 4E; Ref. 117).

Additionally, the dynamical systems view predicts that different computations are performed in different dimensions (i.e., different combinations of latent factors) so that they can be performed independently. To illustrate this point, we return to the distinction between movement preparation and execution. Many neurons are active during both movement preparation and movement generation, and representational frameworks struggle to articulate how a system should organize activity to perform preparation-related computations without inadvertently triggering movement. From the dynamical systems perspective, neural population activity during preparation can be shown to avoid dimensions that trigger movement (114, 115). A similar organizational structure is observed when performing cortically dependent versus non-cortically dependent movements (118), when integrating sensory feedback during movement (119), or when moving the contralateral versus ipsilateral arm (120).

Some aspects of the dynamical systems view may seem trivially true. For example, the principle that future neural activity is somewhat predictable based on the current state may seem obvious for two reasons. First, the brain is a recurrently connected network, so current activity should influence future activity through these recurrent connections. Second, especially in the case of prepared movements, activity during preparation should be informative about activity during execution, so there should exist some (potentially complex) relationship between preparatory state and activity during execution. What is perhaps more instructive about the dynamical systems view is that mathematically tractable descriptions of dynamics can be used to relate preparation and movement execution (98). Furthermore, these descriptions generalize across a wide variety of conditions, to the point where dynamics described for a few movement conditions can be used to predict the system’s behavior across many others (113, 121, 440). Finally, the dynamical systems approach reveals structure in the population dynamics that can predict single-trial behavior on millisecond timescales, which has the potential to improve BMI performance (113, 122, 440–442).

The dynamical systems view has revealed remarkable structure in the population activity of motor cortical neurons associated with reaching movements, yet questions remain as to its generality to hand control. For example, similar dynamical structure is seen during individual finger movements (123), but no such structure is observed in the activity associated with grasping movements (124). Furthermore, the dynamical systems view is not necessarily inconsistent with the representational view. Indeed, one view describes the structure of population activity and the other describes its relation to movement. Better understanding how these two perspectives intersect constitutes a fruitful line of current inquiry (125).

5. DECODING: TRANSLATING INTENTION INTO ACTION

As summarized above, the brain of an able-bodied individual comprises extensive circuits whose principal role is to give rise to movement. In individuals with tetraplegia, these circuits are largely intact but are no longer connected to the spinal circuitry that implements intended movements. To the extent that a relationship between motor intent and patterns of neuronal activity is identified, neural activity patterns can be translated into control signals for external devices or for FES systems. The neural activity patterns that constitute the input to the decoder can take a variety of forms, ranging from the well-isolated responses of individual neurons to the power of extracellular voltage signals in specific frequency bands, reflecting the aggregate activity of large populations of neurons. Control strategies typically fall into one of two broad classes: continuous control, in which the decoder’s output is a continuous variable (e.g., velocity or position) computed on a moment-by-moment basis, and discrete control, in which neural activity is classified to choose between a discrete set of actions (e.g., reaching toward one of several targets, implementing a specific grasp). Some BMIs leverage hybrid strategies in which certain functions are controlled in a continuous fashion (moving to the location) and others are controlled discretely (target selection or grasp). We discuss each aspect of this process and the different control strategies in turn.

5.1. Intracortical Neural Signals for Prosthetic Control

Extracellular electrodes can monitor spiking responses from several surrounding neurons, so early BMIs focused on separating action potentials into groups based on the neuron that generated them (“spike sorting”) and using sorted spikes as features for decoding. The emphasis on using sorted spikes as inputs to BMI decoders was initially motivated by the characterization of the response properties of single neurons, which entailed the identification of a systematic relationship between spiking and movement parameters. Unfortunately, spike sorting is surprisingly difficult. Even with stable action potential waveforms, sorting spikes requires optimizing many parameters to define each waveform template and classify spikes from different neurons. To automate this process is challenging and has been the focus of considerable research (126, 127). Spike sorting becomes much more challenging when waveforms are changing over time, as they tend to do; using a fixed waveform template thus often leads to misattribution of spikes. One approach to address this problem consists in dynamically adjusting the template to account for changes in the waveforms, a hard problem even in an offline setting, but several promising solutions have been proposed (126, 127). Although real-time, adaptive spike sorting may be feasible, this approach inevitably fails to maintain stability as neurons completely disappear from the electrode’s recording range and others enter the range (128–130).

However, achieving a high-performance BMI may not require spike sorting. Indeed, alternative signals provide comparable information to isolated single units without the overhead, complexity, and instability inherent to spike sorting. In fact, most of the highest-performing BMIs over the last decade have relied on multiunit activity (MUA), the aggregate activity of neurons that happen to be close enough to the electrode that their spikes are detectable though not necessarily distinguishable. Typically, a voltage threshold is set and all voltage deflections that exceed this threshold constitute the input to the BMI decoder (FIGURE 6A). MUA shares many of the same features of spiking activity: it is informative about reaching and grasping in monkeys (131–134) and humans (135, 136) and can even be used to extract neural population dynamics (113, 122, 123, 137). However, MUA has additional advantages over sorted spikes; it can often be obtained from electrodes that do not show evident spikes (38, 131, 133) and it is more resilient to the challenges associated with changes in spike waveform over time. Furthermore, hardware optimized to extract MUA can acquire and filter signals with substantial power savings over spike sorting (138–140), a clear advantage for fully implantable, battery-powered BMIs. Accordingly, MUA has rapidly become a standard signal source for online BMI control.

FIGURE 6.

Common signal processing and decoding architectures. A: electrical activity from multichannel neural recordings is filtered, and action potentials are extracted by a simple voltage thresholding operation. Multiunit action potentials are then binned into discrete counts. B: the Wiener filter consists of a set of decoder weights for each neuron across multiple time points. These weights are multiplied by a window of activity and summed to produce the decoder’s output for a given movement variable. C: the population vector algorithm (PVA), single-timestep optimal linear estimation (OLE), and the standard Kalman filter can all be formulated as separable transformations for dimensionality reduction and temporal smoothing. All multiplications shown are element-wise.

Another potential neural signal that can be leveraged in a BMI is the local field potential (LFP), which reflects the aggregate synaptic activity of local populations of neurons, including the spiking activity of neurons that may be too distant to yield discernible waveforms. LFPs are extracted by filtering the time-varying voltage trace captured by each electrode with a low-pass filter. Typically, LFPs are then divided into distinct frequency bands, which differ from study to study. The high gamma bands (e.g., 70–300 Hz) are a particularly rich source of movement-related information, perhaps because this band preferentially reflects spiking activity (141), even in the absence of distinguishable action potential waveforms (38, 131). An alternative feature of the LFP that can be leveraged for control is the local motor potential (LMP; also referred to as the movement-evoked potential), a slow, complex waveform that exhibits substantial directional tuning around movement onset (131, 142–144). In parietal cortex, LFP power at the low frequencies (<20 Hz) exhibits marked changes around the time of movement onset (145, 146), a feature that can be used to trigger a movement (145). Other prominent features have received less experimental attention for closed-loop control. For example, the beta band—which typically spans the range from 15 to 40 Hz, though it differs widely from study to study and may differ between monkeys and humans (113, 143, 147–149)—is typically suppressed around movement onset in motor cortex (147, 149) and may thus be useful for detecting state transitions between stopping and moving. Although the relation between beta oscillations in intracortical activity and those detectable at the level of less invasive interfaces (electrocorticography, electroencephalography) is largely unexplored, recent work suggests that the time course and magnitude of these oscillations can be extracted on single trials with high fidelity from intracortical recordings (113).

5.2. Decoding Architectures for Continuous Control

Continuous control allows for the most versatility in that, in principle, the user is endowed with a vast array of possible actions, defined by the number of degrees of freedom (DOFs), which denotes the number of independently controlled parameters, for example joints or muscle stimulation channels. However, the greater the versatility of the effector—the more DOFs it comprises—the more reliable and complex the neural signals used for control need to be. Over the last few decades, a variety of approaches have been developed to achieve continuous control. Early success was achieved by leveraging the direction tuning of motor cortical neurons, described above. The idea is that, given that individual neurons fire in proportion to the match between intended movement direction and their preferred movement direction, one can infer from the activity across a population of such neurons the intended direction of movement (88). In brief, each neuron “pushes” the decoder’s output in favor of its own preferred direction, with the push proportional to its normalized firing rate. In other words, the decoder interprets strong activation in a given neuron as strong evidence that the intended movement is in that neuron’s preferred direction. A straightforward method to infer intended direction is by tallying the weighted pushes across the population, a strategy called the population vector algorithm (PVA; Refs. 150–153). To calibrate the PVA, a vector containing each neuron’s preferred direction is first estimated from neural responses recorded during actual or intended movements (see sect. 5.4). To decode the intended movement at each timestep (where a timestep might span tens to hundreds of milliseconds), then, the vector of preferred directions is scaled by the neurons’ normalized firing rate and summed to produce a population vector that represents the decoder’s output. The standard PVA has two clear drawbacks: First, if the neurons’ preferred directions do not uniformly sample the dimensions being controlled by the BMI, the PVA’s output is biased (154, 155). Second, decoding relies on neural data from just a single moment in time, which can result in unreliable decoding of intended movement.

An alternate decoding approach is optimal linear estimation (OLE), which, unlike the PVA, does not solely rely on assessing direction tuning at the level of individual neurons. Rather, the approach uncovers the best linear relationship between movement variables and neural responses across the entire population (154–157). A standard implementation of OLE consists of decoding activity over single time points after smoothing the firing rates (single-timestep OLE; Refs. 20, 122, 154, 158–160). One variant of single-timestep OLE first estimates the neurons’ tuning properties (20, 154, 158, 159) similarly to the PVA but with the advantage of correcting for potential asymmetries in the distributions of preferred directions. A potentially more powerful application of OLE uses multiple timesteps of neuronal activation to generate an output, with each neuron ascribed a different weight at each time point (FIGURE 6B) (22, 122, 157, 161–165). This multitimestep approach, referred to as Wiener filtering, has been shown to yield robust closed-loop cursor control in monkeys (122). Wiener filtering can also be augmented to include a nonlinear output stage (the Wiener cascade), which improves decoding of muscle activity for FES-based BMIs (166, 167). The highest-performing BMIs for robotic arm control by people with paralysis have relied on single-timestep OLE (20, 160).

Another popular linear decoding approach is the Kalman filter (162, 168, 169), a standard engineering approach frequently used in guidance and control applications. In BMIs, Kalman filtering aims to improve decoding of motor intent from neural activity by leveraging an additional source of information: the dynamics of the movements themselves. In the real world, kinematics and other movement variables do not arbitrarily change from timestep to timestep but rather are smooth and change gradually. The Kalman filter leverages this smoothness through a recursive approach, in which prior knowledge of the system’s state (previously decoded values of the movement variables) is integrated with noisy measurements of the neural activity at the current timestep to produce an updated estimate of the system’s state. In practice, most BMIs based on kinematic Kalman filters model both the movement dynamics themselves (the state transition model) and the mapping from movement variables to neural activity (the observation model) as linear transformations (162, 169). In its simplest incarnation, the Kalman filter can be reduced to two distinct linear transformations: exponential smoothing in time and a dimensionality reduction stage that maps neural activity onto the decoded movement variables (FIGURE 6C) (170). In this sense, the standard Kalman filter is not drastically different from the single-timestep OLE, except that the degree of smoothing in the Kalman filter is optimized to match the dynamics of the movement variables being decoded, rather than chosen by hand or separately optimized. The highest-performing clinical BMIs for communication and cursor control applications to date have used the standard Kalman filter (35, 37, 38). Several variants such as switching (171), unscented (172), Gaussian process, and discriminative (173) Kalman filters aim to relax the linear and Gaussian assumptions of the state transition and observation models. Additional approaches have included particle filters (172, 174, 175) and point process filters (64, 176–179), which are similar in spirit to the Kalman filter but use more flexible modeling assumptions that may better match neural and behavioral data.

Given the exciting progress in machine learning and artificial intelligence over the past decade, neural networks are emerging as a particularly promising avenue for improving BMI performance. Although early tests of BMIs using neural network decoders did not show marked advantages over their linear counterparts (161, 164), more recent tests show promise (113, 130, 180–182, 440–443). Indeed, neural network-based BMIs yield comparable closed-loop performance as do state-of-the-art linear approaches (130, 183) but offer the potential for dramatically improved robustness to changes in recording conditions over time (130). In closed-loop simulations, neural network decoders have also achieved substantial improvements by incorporating information about the task being performed to dynamically augment movement trajectories (184). Additionally, although the performance of linear decoders has steadily improved through specific optimizations for closed-loop control (discussed in sect. 6.1), such efforts have not yet been applied to neural network-based BMIs and may result in substantial performance improvements.

Although BMI decoders based on strictly representational models have yielded remarkable BMI control, efforts are underway to examine whether incorporating neuronal dynamics can further improve performance (reviewed in Ref. 108). As detailed above, a main tenet of the dynamical systems view is that neuronal responses in motor cortex evolve over time in orderly and predicable ways and these dynamics do not necessarily relate in straightforward ways to the movement variables. However, taking this underlying structure into account may allow decoders to better map neuronal activity onto movement parameters. One strategy consists in extending the Kalman filtering approach, which traditionally models the dynamics of the observed movement variables, to model the unobserved dynamics of the neural population [the neural dynamical filter (NDF); Refs. 108, 122, 185, 186]. In monkey experiments, using the NDF to model neural population dynamics led to improved online performance over standard Kalman filtering, single-timestep OLE, or Wiener filtering (122). The NDF is based on the standard Kalman filter and thus relies on simple linear models of neural population dynamics. Further decoding improvements may be achieved through nonlinear modeling, as demonstrated in offline settings using neural networks (108, 113, 121, 182, 440–442).

5.3. Decoding Architectures for Discrete Variables

A fundamentally different approach consists in decoding one among a discrete set of actions—reaching to a specific end point, move versus stop, click versus no click, pinch grasp versus power grasp versus no grasp—from neural activity. This strategy consists in first building a library of patterns of neural activity associated with each action. Detection of a stored pattern then triggers the associated action. One challenge of discrete decoding is the inherently dynamic nature of neural activity. Because neural activity related to a specific action evolves over time, the corresponding activity pattern for that action will depend on when activity is sampled. A straightforward and effective strategy to sidestep this challenge is to pace the approach by instructing the user to initiate an action at particular times and decoding at those specific times (100, 187). In one study employing this strategy, monkeys planned movements to one of a fixed set of targets at prespecified time points controlled by the experimenter (187). The decoder consisted of a statistical model associating patterns of neuronal activity to each potential target. On any given trial, an action was selected if the observed activity more closely resembled activity patterns associated with that action than patterns associated with other actions. Although this strategy resulted in high performance, a key drawback of the fixed-pace approach is that it assumes knowledge of when the user wants to use a device and is therefore only applicable to tasks that can be paced. This approach can be extended to user-defined self-pacing by implementing a separate state estimator, operating on a sliding window of neural activity, that determines transitions between baseline, planning, and execution states (188).

A common approach to discrete decoding is to use linear discriminant analysis (LDA), a dimensionality reduction technique like principal components analysis (PCA). Unlike PCA, however, LDA does not attempt to find reduced dimensions that capture as much of the variability in the neural activity as possible. Instead, LDA maximizes the distance between patterns of neural activity associated with different actions within a low-dimensional space and then finds boundaries within that space to best separate activity patterns from different classes. LDA has been used to select targets during typing (34, 36, 149), to detect movement and stopping states (189, 190), and to classify movement goals (17).

An alternative architecture for discrete classification is the K-nearest neighbors (KNN) algorithm (86). Like LDA, KNN relies on the notion that patterns of neural activity that belong to the same class are close to each other in some multidimensional space. However, KNN classifies a new pattern by finding the K most similar patterns (nearest neighbors) in a training data set, where K is a parameter chosen by the user. The new pattern is assigned to the class that has the most neighbors. KNN is a nonparametric method that makes fewer assumptions about the statistical properties of the data than LDA. KNN has been used to decode hand shape from posterior parietal cortex as a person with paralysis played rock-paper-scissors (and its extension, rock-paper-scissors-lizard-Spock) (86). Additionally, combining recurrent neural networks with KNN classifiers to decode imagined handwriting achieved state-of-the-art communication performance (4).

A common architecture to continuously decode discrete actions is the hidden Markov model (HMM; Refs. 38, 191, 192). HMMs are probabilistic models that estimate the probability of being in one of a set of states at any given time point. HMMs for neural state classification have two components: an “emission” model, which determines the probability of neural observations given each discrete state, and a “transition” model, which describes the probability of transitioning between states. To produce an estimate of the current state, HMMs can weight two sources of information: the likelihood of transitioning into each state based solely on the identity of the previous state and the likelihood of being in each state based on how consistent the observed neural activity is with the emission model for that state. The state transition model helps the HMM avoid transitions caused by spurious fluctuations in the neural activity. With these two components, HMMs are broadly analogous to Kalman filters, except that the unobserved variable being modeled is a discrete state rather than a continuous movement parameter.

5.4. Calibration of Biomimetic Decoders

Two broad strategies have been traditionally used to build BMI decoders: Biomimetic approaches attempt to mimic the brain’s interaction with the native limb, whereas adaptation-based approaches implement a quasi-arbitrary mapping between neural activity and movement and capitalize on the brain’s ability to adjust its activity patterns based on feedback during online BMI control. Biomimetic strategies have achieved the highest-performing BMIs to date, whereas nonbiomimetic strategies have revealed insights into how and how much the brain can learn (reviewed in sect. 6.2). Biomimetic and adaptation-based strategies are not mutually exclusive and are likely to ultimately complement each other (193).

To calibrate a biomimetic decoder in an able-bodied subject, e.g., a monkey, neuronal activity is monitored as the animal observes preprogrammed movements of a cursor or robotic arm (and keeps its arm still) or moves its arm to control the cursor or perform a manual task (194). Recorded neural activity is then mapped onto the observed or performed movements with one of the decoding models described above (PVA, OLE, Kalman filter, etc.). The two calibration approaches have different advantages and disadvantages with regard to clinical relevance (194, 195): Requiring the monkey to keep its arm still prevents it from receiving somatosensory feedback (which would not be present in paralyzed subjects). However, it also forces the monkey to avoid generating patterns of neural activity that would normally drive movements (paralyzed subjects have no such constraint). Allowing the animal to move its arm features more natural control signals but also sensory feedback. In studies with paralyzed participants, decoders are generally built from neuronal responses collected as the participant observes movements of the end effector (cursor, virtual robotic arm, or physical robotic arm) and attempts to mimic the observed movements.

Although observation-based and movement-based calibration with intact animals each have their advantages and disadvantages, the two strategies may ultimately be equally successful. Indeed, the two calibrations were implemented with a partially paralyzed human participant who retained movements of her fingers, and the two calibration procedures yielded similar performance (35, 38), echoing results with monkeys (196, 197). In fact, monkeys often choose to stop moving during closed-loop brain-computer interface (BCI) control, even when the underlying decoders were trained with physical movements (161, 163, 168), suggesting that neuronal activity during physical movements and attempted or imagined movements show similarities that can be leveraged by BMI decoders (197).

6. FROM OFFLINE DECODING TO ONLINE PERFORMANCE

Over the last decade or so, the quality of the prosthetic control achieved via brain signals has steadily improved. For example, in cursor control and communication, clinical BMIs have achieved two- to threefold improvements in performance (35, 38). Surprisingly, decoding architectures have remained largely unchanged over this period. What factors, then, have driven the performance increase? The answer lies in a shift in focus from offline algorithmic innovations to online BMI control, in which intended movements are decoded in real time and the user can observe the decoded movements. Naively, one might expect the decoding problem to generalize from offline to online: decoders that are best able to reconstruct movements based on neural activity offline would be expected to achieve the highest performance online as well. In practice, however, offline reconstruction accuracy has limited correspondence with online decoding performance. For example, decoding from a population of neurons with an asymmetric distribution of preferred directions leads to biased decoding with the PVA relative to single-timestep OLE, as noted above. Although these biases are prominent in offline reconstruction, however, they often have little impact on online performance because users are able to adapt to systematic errors in the decoded movements (154, 198). Similarly, whereas offline reconstruction can be improved by binning spikes with large windows (e.g., 150 ms) to reduce noise, smaller bins (e.g., 20 ms) yield higher online performance by allowing the user to make rapid corrections (199), which has led to an emphasis on real-time BMI architectures that minimize the latency between neural signals and movement of the end effector (35, 196).

Accordingly, selecting algorithms for online decoding based on offline reconstruction accuracy is ill advised. As a case in point, in an early comparison of Wiener and Kalman filters for cursor control—both optimized offline—the Kalman filter yielded higher online performance and faster movements (162), which likely cemented this decoder’s place as the performance benchmark (35, 149, 196, 200). Subsequent analysis revealed that the advantage of Kalman over Wiener filters may have been an artifact of offline optimization: Wiener filters that can leverage long windows of neural data may introduce long delays between changes in neural signals and changes in movement output, which would limit the speed of movement initiation and the ability to perform rapid feedback corrections (122, 165). When Wiener filters are properly optimized by restricting window lengths, they outperform Kalman filters and single-timestep OLE (122).

6.1. Optimizing Decoders for Closed-Loop Performance

A key insight driving the recent improvements in BMI performance has been the recognition that the patterns of neural activity that are engaged during online BMI control may be substantially different from the patterns engaged when subjects move their native arms [monkeys (153, 196, 201, 202)] or when they attempt (or imagine) movements to follow preprogrammed movements of an effector [monkeys and humans (35, 200, 203)]. In light of these differences, decoders are often initially calibrated through an offline optimization process (as described in sect. 5.4) and then recalibrated using data collected during online control. One strategy—known as orthoimpedance—consists in providing the subject with assistance during online control, such that the decoder’s output is modified by attenuating components of the control signal that do not point toward the experimenter-defined target (9, 20, 24). Attenuating these error components effectively serves as an online assist to facilitate BMI control based on an initial, potentially flawed decoder. The neuronal activity collected during this assisted online control then serves as a training data set to produce a recalibrated decoder. An alternative and complementary approach—dubbed intention estimation—attempts to infer the subject’s intent from neuronal activity monitored during online control (203). When a BMI is used to navigate between predefined targets, for example, in a reaching or a cursor control task, one typically assumes that the subject intends to move the effector straight to the target. During online control, the actual decoded output may be noisy, resulting in a mismatch between intended movement and decoded movement. However, the inferred intention applied post hoc can be used to produce a recalibrated decoder, leading to a substantial improvement in subsequent online performance (35, 196, 200). Intention estimation can be implemented in a variety of ways (35, 153, 196, 200, 203), but these lead to similar performance (203). Perhaps counterintuitively, intention estimation has been shown to provide benefits even when building decoders based on kinematic and neuronal activity collected during native arm control (in monkeys) (201). That is, a better online decoder can be achieved by largely ignoring the actual movements of the native arm and instead calibrating the decoder based on inferences of what the monkey “intended” to do with its arm. During native arm control, the arm does not necessarily move directly to a target but instead may take a curved trajectory or move off the target axis because of motor noise. Iteratively updating a decoder during online control can achieve high performance within minutes, even if the decoder begins from an arbitrary or uninformed starting point (173, 204, 205). After a short period, the decoder stabilizes, having achieved a solution that accurately infers user intent (205).

In online control, BMIs face a trade-off between the ability to produce rapid movements and allow fast corrections and the ability to move precisely and hold still when needed. Two parameters that define this trade-off, and thus the overall feel of the BMI, are the degree to which neuronal activity is smoothed, which determines whether the decoder’s output changes rapidly or gradually in response to changes in neural firing patterns, and the gain or speed scaling, which determines the conversion between changes in decoder output and changes in the effector movements (203). Less smoothing allows for rapid changes in the decoder’s output but leads to increased sensitivity to small fluctuations in neural activity that may be noise. Higher gains allow the user to quickly move an effector across a workspace but may also amplify noise, thereby making it difficult to hold the effector still (206). Although higher gain and lower smoothing allow for fast movements and quick adjustments, lower gain and higher smoothing lead to more stable estimates of intended movements and improved stopping. Gain and smoothing can be adjusted in a principled manner based on task demands and user-specific latencies, leading to improved performance (207). Additionally, some aspects of this trade-off can be addressed by explicitly identifying brain states associated with stopping (190), scaling neuronal output nonlinearly to suppress baseline fluctuations and enhance the dynamic range of the decoded output (207), or attenuating speeds when jitter is detected (208). Finally, neural population dynamics may provide more principled methods to smooth data. For example, recurrent neural networks can model the rich and complex temporal dynamics of neural activity, and this knowledge can be used to reject features of neural activity that are inconsistent with the learned dynamics. In offline settings, this denoising approach has been shown to enable substantial improvements in decoding accuracy over smoothing using just a single-timestep OLE decoder (113), while also avoiding the fixed-smoothing restrictions imposed by standard Kalman or Wiener filters.

A challenge that currently limits online performance is the inability to collect sufficient data to calibrate decoders that generalize. Indeed, the relationship between neural activity and behavior can be context and task dependent (209–212). For example, the relation between neural activity and kinematics can change depending on whether objects are present in the workspace (209) or even on the identity of the objects (212). One approach to achieve a robust decoder is to calibrate across different task contexts (160) over multiple sessions (113, 130), or even across multiple subjects (213).

6.2. Neural Adaptation and Learning

As outlined above, BMIs often adopt a biomimetic strategy that exploits the way the brain naturally controls the limb. A separate category of approaches places less emphasis on the match between an initial decoder and native arm control and instead leverages adaptation of the nervous system—and, potentially, parallel adaptation of the decoder—to improve performance over time. In early BMI studies with monkeys, experimenters would often introduce a context switch after decoder calibration (e.g., removing a manipulandum that the monkey used during calibration) before beginning online control, which required that the monkey find an alternate strategy to generate the patterns of neural activity required to control the decoder. Despite this, monkeys were able to learn to perform the task (100, 153, 161, 214).

More recently, BMIs have been used to probe the capabilities, limitations, and timescales of neural adaptation. In one example, monkeys learned to control an on-screen cursor using a fixed but arbitrary mapping (215). Over the course of days to weeks, the monkeys gradually shifted from being completely unable to use the cursor to achieving near-perfect performance. During this time period, the relation between individual neurons’ activity and the cursor’s movements (i.e., the neurons’ tuning) gradually stabilized. The monkeys could also learn to use a second arbitrary decoder over the course of days, while maintaining performance with the first decoder. This work provides compelling evidence of the flexibility of the motor cortex to learn to generate and consolidate new patterns of neural activity.

A recent series of studies invoked the concept of neural manifolds to systematically interrogate constraints on the brain’s ability to generate new patterns of neuronal activity (216–218). Monkeys were trained to use decoders that were built by first mapping neural activity onto a low-dimensional manifold and then mapping this manifold activity onto the movements of a cursor. The experimenters then perturbed the relation between neural activity and cursor movements in two systematic ways: 1) perturbations that only affected the second mapping, i.e., that preserved the mapping from neural activity onto the manifold but changed the relation between manifold activity and cursor movements (within-manifold perturbations), and 2) perturbations that required the monkey to generate patterns of activity that were not contained in the neural manifold (outside-manifold perturbations). Within a timescale of minutes to hours, monkeys could readily learn to control the cursor after within-manifold perturbations but were much slower to learn outside-manifold perturbations (218), pointing to manifold structure, presumably reflecting the inherent circuitry of the nervous system, as a fundamental constraint on the brain’s ability to produce patterns of neural activity. Furthermore, even for within-manifold perturbations, the monkeys did not reshape their activity within the neural manifold itself. Instead, monkeys continued to generate the same patterns of activity but associated this activity to correspond to different intended movements (216). Subsequent studies of long-term learning demonstrated that outside-manifold perturbations could be learned over the course of days or weeks, and this learning corresponded to the emergence of new patterns of activity (217). These studies demonstrated that the brain can much more readily learn to produce some patterns of activation than others and that dimensionality reduction techniques can be used to distinguish easy from hard patterns.

7. LONGEVITY AND STABILITY

Intracortical BMIs necessitate a craniotomy and implantation of electrode arrays in the brain. For that reason, the system should be sufficiently robust to last through the user’s lifetime so that repeated surgeries are not necessary. Although compelling BMI performance has been achieved using intracortical electrodes that last for years with minimal complications (219), neural interface recording quality and stability face several challenges. The principal challenge is that the ability to monitor neuronal activity degrades progressively—and sometimes critically—because of tissue reaction, degradation of the implant, or failure of the connector (220, 221). Electrode impedances decrease on the timescale of months, as do the number and amplitude of observed action potentials (221). In addition to changes in recording quality, electrodes also exhibit changes in the specific neurons being monitored within and across days, due mostly to small movements of the electrodes relative to the surrounding tissue (128–130).

As noted above, MUAs and LFPs, which reflect the aggregate activity of neurons around the electrode, are far more stable than individually sorted spikes and are often accessible on electrodes with no discernible action potential waveforms. These sources are thus more robust to degradations in signal quality. Hybrid decoding strategies that leverage both signal types have demonstrated high-performing control from intracortical arrays with signal quality that may once have been considered too poor for BMIs (35, 38, 144) and may be helpful in extending the lifetime of BMIs.

A commonly held view is that implanted devices with less spatial precision [e.g., electrocorticography (ECoG)] may have advantages over intracortical interfaces, in terms of either stability or longevity. Notably, however, the performance of intracortical interfaces, even 2–3 yr after implantation, remains severalfold higher than the highest-performing alternatives (20, 35, 38, 160). As described below, the robustness of intracortical BMIs continues to improve, and emerging methods, described next, may lead to ever more robust control.

8. DECODING STRATEGIES TO MITIGATE INSTABILITIES OF THE NEURAL INTERFACE

Standard practice to compensate for instabilities in the signal has been to recalibrate the decoders, once or several times per day (5, 9, 18, 20, 35, 37, 38, 160). However, this intermittent recalibration not only interrupts device use but also typically requires third-party intervention. In addition, frequent changes in the decoder may require the user to adapt to the change, as if suddenly having to use a different tool.

Multiple strategies have been developed to mitigate instabilities and avoid supervised recalibration (222). One approach consists in automatically and continuously recalibrating the BMI decoder during ongoing device use (36, 223–227). For example, recent “semisupervised” approaches have been developed to recalibrate decoders on the fly, using a retrospective analysis of data collected during the subject’s normal use of the BMI (36, 225). This process mimics the intention estimation process described above. For example, in a setting with distinct targets, the neural activity preceding movement to a given target likely reflects the subject’s intention to move toward that target. This knowledge can be used to update the decoder, much like the supervised online recalibration procedures described above. In studies with people with paralysis, semisupervised recalibration represents the state of the art, conferring stability across months of BMI use. However, the semisupervised approach has been applied only to simple cases in which the user’s intent can be deduced post hoc and may not scale to more complex, naturalistic settings.

A promising and potentially complementary approach consists in training neural network decoders with many sessions distributed over extended periods of time, during which the decoder is exposed to a wide variety of recording conditions (130). This approach is based on the idea that, although neural recordings may be unstable, the specific configuration of observed neurons at a given moment may be similar to a previously observed condition. Neural networks are particularly valuable for such applications, as they can learn and simultaneously carry multiple mappings between neural activity and intended movements. In experiments with monkeys, this approach has yielded impressive improvements in robustness, maintaining performance under conditions that would lead to catastrophic failure with standard decoders. One challenge will be to adapt such an approach for clinical applications, where collecting such large, supervised data sets constitutes a substantial burden.

A separate class of approaches aims to leverage the stability of the manifolds and dynamics underlying neural population activity (FIGURE 7) (108, 113, 186, 222, 228–232). As described above, manifolds are a property of the underlying network rather than any individual recorded neurons and thus do not require maintaining recording of a specific set of neurons. Importantly, manifolds exhibit a consistent relationship with behavior across month- to year-long timescales that is independent of neuronal turnover (108, 113, 233). Thus, decoders trained to relate manifold activity to behavior may not require recalibration. A separate challenge relates to manifold alignment, i.e., mapping changing neural activity onto the stable underlying manifold to maintain robust decoding (222, 228). Although this process is straightforward if a supervised data set is available (i.e., when both the neural activity and the subject’s behavior have been monitored), a supervised recalibration step is necessary for each realignment. In that case, since supervised data are available, there may be little benefit to realignment over simply calibrating a new decoder, save the possibility of achieving a high-performing decoder with less data through realignment. Instead, several approaches are being developed to periodically update the neural activity-to-manifold relationship in an automated, unsupervised fashion (222, 228–232). One stabilization approach, recently tested with monkeys, exhibited stable online control in the presence of both spontaneously occurring and imposed instabilities over the course of a 5-day period (229). Without stabilization, the monkeys were unable to adapt to these instabilities, indicating that manifold stabilization can overcome progressive changes in the neuronal sample that would otherwise substantially degrade BMI performance.

FIGURE 7.

Leveraging manifolds and dynamics to improve brain-machine interface (BMI) stability. A: recording locations with a 24-channel v-probe in monkey primary motor cortex (M1) from 44 recording sessions spanning 5 mo. CS, central sulcus; Arc. sp., arcuate spur; PCd, precentral dimple. B: supervised alignment allowed recovery of a single neural manifold, with consistent trajectories across 5 mo. Each trace is from a different session, with colors for different 2-dimensional reach directions (reaches shown at bottom). C: decoding performance with/without manifold alignment. Each dot/trace represents an individual recording session. Decoding from observed activity with the 24-channel probe yielded poor estimates of hand velocity. The neural manifold produced by LFADS achieved substantial improvement in accuracy for all 44 sessions, even though a single fixed linear decoder was used for all sessions. GPFA, gaussian process factor analysis; LFADS, latent factor analysis via dynamical systems; VAF, variance accounted for. A–C taken from Ref. 108, with permission from Journal of Neuroscience. D: using a simple supervised method to align Utah array data to an underlying manifold decoder across 2 yr of data (blue) consistently maintains decoding performance to within 5–10% of a daily-recalibrated decoder (gray). CCA, canonical correlations analysis. Adapted from Ref. 233, with permission from Nature Neuroscience. E: manifold stabilization can maintain and recover online BMI performance without supervised recalibration. Success rate (top) and acquisition times (bottom) over the course of a 5-day experiment in which synthetic instabilities were periodically introduced. NC, not computed. Adapted from Ref. 229, with permission from Nature Biomedical Engineering.

9. PERFORMANCE METRICS FOR BMIs

The field of intracortical BMIs has been steadily moving beyond proof-of-concept demonstrations and toward clinical translation. At the same time, large industry efforts—Neuralink, Paradromics, Synchron, Kernel, Facebook, Iota, and Microsoft, among others—are developing BMIs that span all levels of invasiveness toward a wide range of commercial applications. Given these trends, the ability to compare performance across research groups and interfacing modalities is essential for understanding the capabilities and relative merits of different approaches and for tracking progress. At present, however, such comparisons are hampered by inconsistent metrics and reporting standards. Thus, a crucial aspect of the field’s maturation is the development of standardized metrics that generalize across experimental paradigms.

Performance metrics generally divide into two classes: functional metrics aim to assess real-world capabilities (as would be important to an end user), and control quality metrics gauge more fundamental aspects of BMI control (as might be useful for detailed comparisons of decoder performance or information throughput). The advantage of functional metrics is that they typically measure performance on real-world tasks and are thus intuitive. They also have multiple disadvantages. One is that they are application specific and thus may not generalize to other task domains. A second is that, by evaluating end-to-end performance of a complex system, it may be hard to pinpoint the key factors driving performance (interface modality, decoding or stimulation strategies, etc.), which could hamper researchers’ ability to build off of previous work. In contrast, control quality measurements allow for fine-grained evaluation of BMI performance, but they are hard to extrapolate to real-world tasks. Thus, control quality measurements may be more useful when comparing BMIs that have similar functional capacity and to disentangle the factors that drive performance.

Regardless of the assessment type, performance reliability is a crucial consideration for any assistive technology yet has often been underemphasized in BMI research. As discussed above, performance might vary substantially across days or even over the course of hours, sometimes necessitating intervention to restore control. As a result, single-time point evaluations, best-case reporting, or protocols that allow intervention from a technician may paint a misleading picture of the potential clinical utility of a BMI, and evaluation standards that emphasize reliability are critical to the field’s translational efforts.

Below, we summarize current and proposed assessment standards and describe examples in which metrics have facilitated rigorous quantitative comparisons across studies and interfacing modalities. We also summarize best practices for quantifying performance reliability.

9.1. Functional Metrics

For BMIs aiming to restore upper limb function, functional metrics have been adapted from standard assessments for patients who have had a stroke or spinal cord injury. One such test is the box and blocks test (237), which provides a rapid but somewhat limited assessment of grasping and object transport. In this test, subjects are tasked with picking up small blocks from one compartment, transporting them over a barrier, and releasing them into another compartment. The task is scored based on how many blocks can be transferred within a minute. The box and blocks test and variants have been used to assess restoration of grasp via brain-controlled FES (238) and robotic arms (6, 160). Another common test is the Action Research Arm Test (ARAT; Ref. 239), which assesses an individual’s ability to undertake several activities of daily living, including grasp, grip, and pinch, as well as gross arm movement. The ARAT has also been used to gauge performance of brain-controlled FES (238) and robotic arms (6, 20, 160). In the latter studies, pinch and gross arm movement tasks were removed, as these were beyond the capabilities of the BMI. A third common test, the Graded and Redefined Assessment of Strength, Sensibility, and Prehension (GRASSP), was developed to assess upper limb impairment for individuals with tetraplegia and includes tests to assess grip strength, sensation, and the ability to generate different grasp types (240). GRASSP has been used to evaluate performance of brain-controlled FES (18, 238).

As noted, clinical tests are often simplified because of constraints of the BMI. For example, fine movements and grasping small or noncompliant objects can be difficult with current robotic prostheses. Tests that require these capabilities are thus eliminated when evaluating the performance of BMI-controlled robotic arms. As BMIs evolve, an additional performance metric will be the degree to which standard clinical assessments can be fully implemented.

As BMIs become more complex, it will be hard to disentangle the factors that contribute to performance improvements using functional measures alone. This can be seen in current communication BMIs (4, 38, 213, 241, 242), where improvements in functional performance (correct words/min) can reflect a combination of better decoding of neural intention and better machine learning language models that add word correction and word or phrase completion to predict a message’s content based on context. The former might reflect progress in overcoming fundamental neural interfacing limitations, whereas the latter is not specific to the BMI. Additionally, complicated machine learning tools can make it easy to “hack” limited functional performance metrics without generalizing to real-world use. For high-DOF BMIs, these same issues will become especially important if the field begins to adopt artificially intelligent controllers—as are becoming common in robotics—so that low-level control details can be offloaded onto algorithms that are largely independent of the brain interface. In such situations, separate control quality assessments may be critical for documenting neurally driven performance gains.

9.2. Control Quality Metrics

A critical component of reaching is the ability to translate the end effector between locations and accurately stop. This aspect of point-to-point reaching movements has been assessed with metrics based on Fitts’s law, which describes how movement times scale for reaching movements of varying difficulties in able-bodied individuals (149, 206, 243). To quantify the difficulty of a given movement, Fitts’s law provides a simple measure, the index of difficulty (ID), which is a function only of the distance to a target and the size of that target. Empirical studies in able-bodied subjects demonstrate that movement times generally increase in proportion to the ID (243). ID values are in units of Fitts’s bits, and performance in point-to-point movement tasks often combines the rate of target acquisition and the difficulty of acquiring those targets to yield an index with Fitts’s bits per second as units. TABLE 1 demonstrates how Fitts’s law can be used to compare performance across BMI cursor control studies with people with tetraplegia (updated from Ref. 35).

Table 1.

Performance comparisons across clinical studies of cursor control

Study	Interface	Motor Impairment Etiology	Task Index of Difficulty, Fitts’s bits	Fitts’s bits/s	Success Rate, %
Gilja et al. 2015 (35)	Intracortical (Utah array)	ALS	2.28	0.93	95–100
Simeral et al. 2011 (149)	Intracortical (Utah array)	Brain stem stroke	2.28	0.27	95
Kim et al. 2011 (444)	Intracortical (Utah array)	ALS	2.15–2.36	0.16	45
Silversmith et al. 2021 (10)	ECoG	Brain stem stroke	∼3	0.49 (best)*	100 (best)
Wang et al. 2013 (377)	ECoG	Spinal cord injury	1.18	0.58**	87 (best)

All studies measured performance in point-to-point movement tasks, allowing comparisons across interfacing modalities using Fitts’s bits per second. Updated from Ref. 35. ALS, amyotrophic lateral sclerosis; ECoG, electrocorticography. *Task did not enforce hold times. **Task required reduced control capabilities and precision relative to the other studies listed, as detailed in Ref. 35. Similar comparisons for studies with healthy monkeys have been published previously [Gilja et al. 2012 (196)].

The advantage of Fitts’s law is that it provides an avenue to quantify performance using simple point-to-point reaching tasks, such as common center-out-and-back reaching paradigms, thereby facilitating comparisons across studies. However, Fitts’s law suffers from several limitations that limit its utility in evaluating BMI performance. A major limitation is that there is no prescribed hold time to consider a target successfully acquired. Thus, in an extreme case, a BMI that quickly and randomly drifted across a workspace could achieve high performance without being under neural control in any meaningful way. Another issue is that BMIs often have particular failure modes (e.g., an inability to reach specific targets), and it is unclear how to factor failures into Fitts’s law-derived metrics. Finally, signal-independent noise—relatively large for slow movements and small for fast ones—typically leads to movement times that do not follow Fitts’s law (206). Because of the latter issue, the assumption that movement times scale linearly with ID may be invalid, which can confound performance comparisons across tasks with different IDs.

An alternative method to evaluate control quality is the grid task (10, 22, 38, 242, 244). In the two-dimensional (2-D) case, the task divides a given workspace into a grid of squares, and users must translate the end effector to the target square and dwell to acquire or use an alternate selection method (such as a neurally decoded “click”). Although the spirit of the grid task is similar to Fitts’s law-derived measures, an advantage is that it enforces a higher degree of control precision because users must avoid incorrectly selecting alternate targets. Thus, grid tasks are replacing Fitts’s law-derived measures for quantifying continuous control quality. Performance on the grid task can be measured with achieved bitrate (10, 38, 242, 244, 246), which increases based on the user’s target acquisition rate and the difficulty of the task (grid resolution). Finer grids can be used to test higher-precision control, and dwell times can be optimized based on control quality to avoid incorrect selections. The use of bitrate as a performance metric facilitates broad comparison, including with commonly used communication tasks for EEG-based BMIs and with alternate tasks such as brain-controlled handwriting (38, 242). TABLE 2 charts comparisons for selected high-performance BMIs spanning various interfacing modalities and tasks (adapted from Refs. 38, 242).

Table 2.

Performance comparisons across clinical studies of communication

Study	Interface	Task	Participant	Motor Impairment Etiology	Avg. Bitrate, bps	Avg. ITR, bps
Pandarinath et al. 2017 (38)	Intracortical (Utah array)	Point and click (grid)	T6, T7, T5	ALS (2), SCI (1)	2.4	2.4
			T5	SCI	4.2*	4.2*
Willett et al. 2021 (4)	Intracortical (Utah array)	Handwriting	T5	SCI	∼6.18**
Silversmith et al. 2021 (10)	ECoG	Point and click (grid)	B1	Brain stem stroke	0.71
Nijboer et al. 2008 (247)	EEG	P300	n = 4	ALS		0.08–0.32
Townsend et al. 2010 (246)	EEG	P300	n = 3	ALS	0.05–0.22	-
Pires et al. 2012 (248)	EEG	P300	n = 14	ALS (7), CP (5), DMD (1), SCI (1)		0.05–0.43
Mainsah et al. 2015 (249)	EEG	P300	n = 10	ALS		0.01–0.60
Vansteensel et al. 2016 (250)	ECoG	Sequential speller	n = 1	ALS		0.21

Performance measurements using a grid task facilitate comparisons to studies with different interfacing modalities and brain-machine interface (BMI) tasks. *Single participant assessed with dense grid. **Bitrate approximated with an empirical conversion factor (242). ALS, amyotrophic lateral sclerosis; CP, cerebral palsy; DMD, Duchenne muscular dystrophy; ECoG, electrocorticography; EEG, electroencephalography; SCI, spinal cord injury; ITR, information transfer rate.

Control quality measurements become more challenging as BMIs scale to increasing degrees of freedom. For multijoint control, quality can be assessed through single-DOF tasks where users position individual joints within prespecified acceptance windows (5). Alternatively, for end effector control, sequence-matching tasks (20, 160) specify target postures that the user must match (end-effector position, orientation, etc.). Task performance can be quantified by completion times and path efficiency (i.e., the ability to achieve a straight-line path to target). These measures are most useful for within-study comparison (e.g., using different decoders or including sensory feedback, etc.). At present, the field lacks standardized metrics to facilitate across-study or across-group comparisons of high-DOF control. Additionally, several important aspects of reaching and grasping, e.g., the ability to make curved reaches in cluttered environments, the ability to track moving targets, etc., are not assessed by these measures.

9.3. Reliability

Reporting standards that assess reliability are critical for assessing clinical utility because of the potential variability of BMI control. An increasingly common practice is to evaluate performance by creating a strict measurement protocol a priori that specifies the sequence of events when performing assessments (149). Such a protocol might specify the number of measurements (e.g., number of evaluation sessions), how the BMI is to be calibrated, what to do in cases of failures, whether the BMI can be recalibrated, and whether and how technicians can intervene. If such protocols are followed and performance on individual sessions is reported rather than summaries, the variability in performance can be evaluated, which greatly increases confidence in reported metrics.

10. SENSORY FEEDBACK FOR PROSTHETIC CONTROL

10.1. The Importance of Somatosensation in Everyday Life

When we interact with objects, we receive a barrage of sensory signals from the hand that convey information about the objects themselves—their size, shape, texture, etc.—and about our interactions with them—contact timing, location, force, etc. (3, 251). Without these signals, our ability to use our limbs, and in particular our hands, would be severally impaired, as has been shown with patients with peripheral neuropathies that have destroyed their somatosensory input, both tactile and proprioceptive. Indeed, these patients exhibit movements that are slow and clumsy despite having intact motor systems and report that all behaviors are effortful, akin to running a daily marathon (252). Even milder sensory deficits, produced for example by desensitizing the fingertips with a local anesthetic, impair our ability to exert an appropriate amount of force on objects (253) or pick up small objects when vision of the contact points is obstructed. To achieve a dexterous prosthesis will thus require restoring not only movement but also the senses of touch and proprioception, given their importance in object interactions. Furthermore, somatosensation is critical to our embodiment, the feeling that our bodies are part of us. Indeed, deafferentation of a body part leads to its disembodiment, and the real or perceived afferentation of an artificial limb can lead to its embodiment (254–257); the pairing of the visual experience of a part of our body touching something with the tactile experience of touching it is critical to the embodiment of that body part. Finally, touch plays a critical role in affective communication. We touch the people we love and seek to be touched by them (258). One important and oft-overlooked consequence of limb function is the loss of the ability to touch loved ones, which only magnifies the psychological sequelae of spinal cord injury (13).

10.2. The Neural Basis of Somatosensation

The palmar surface of the hand is innervated by ∼17,000 nerve fibers that can be categorized into three distinct classes, each sensitive to different aspects of skin deformation (2, 259). Some nerve fibers, termed slowly adapting, primarily track the time-varying depth of indentation, whereas others, termed rapidly adapting, respond primarily to changes in indentation. Most tactile nerve fibers can only be activated when a small patch of skin—their receptive field (RF)—is touched; the only exceptions are Pacinian afferents, whose RFs can span multiple digits and sometimes most of the hand. Proprioceptive signals are carried primarily by nerve fibers that innervate receptors in the muscles—spindles—and fibers that innervate receptors in the tendons—Golgi tendon organs (2). Tactile and proprioceptive signals from first-order afferents are transmitted to the cuneate nucleus (CN), located in the brain stem, which in turn sends projections to the ventroposterior nucleus (VPN) of the thalamus. VPN is the last stage of processing before cortex, where somatosensory signals undergo several more stages of processing (1). All tactile and proprioceptive information (thus excluding thermoreception and pain) is carried in the activity in these neural structures, but tactile signals remain segregated from proprioceptive ones until intermediate to late stages of cortical processing.

A key principle underlying somatosensory representations is that of somatotopy (FIGURE 8). That is, neurons that respond to stimulation of the hand are adjacent to neurons that innervate the arm, which in turn are near neurons that innervate the trunk. On the other side of the hand representation is the face representation. Even within the hand representation, the digits are laid out systematically, with the thumb lateral and anterior to the index, itself lateral and anterior to the middle finger, etc. (260). Proprioceptive representations in cortex are also organized somatotopically and generally follow their cutaneous counterparts along the central sulcus (261).

FIGURE 8.

Somatosensory neuraxis with possible targets of a neural interface for sensory feedback. Top left inset: somatosensory cortex is organized somatotopically; the body map in somatosensory cortex is much more clearly defined than that in motor cortex. Top right inset: somatosensory cortex comprises 4 cortical fields. Area 3a is proprioceptive, areas 3b and 1 are cutaneous, and area 2 exhibits both response properties.

10.2.1. Subcortical structures

Little is known about the response properties of neurons in the cuneate nucleus (CN), the first recipient of somatosensory signals from the periphery. Traditionally, the CN has been described as a passive relay station for afferent signals, where no processing or even convergence occurs. Some recent evidence from macaques suggests otherwise, however (262, 263). Indeed, individual neurons seem to receive convergent input from multiple classes of nerve fibers and exhibit response properties intermediate between those observed in the nerve and those observed in cortex. The same seems to hold for neurons in VPN, but the neural representations in CN and VPN have not been systematically compared. Both CN and VPN are organized somatotopically (264–266).

10.2.2. Somatosensory cortex

Located in the postcentral gyrus, somatosensory cortex comprises four distinct cortical fields: Brodmann’s areas 3a, 3b, 1, and 2 (1) (FIGURE 8). Although these four cortical fields are often referred to collectively as primary somatosensory cortex, only area 3b meets the criteria of a primary sensory area in terms of thalamic input and cytoarchitecture (267). Neurons in area 3a exhibit exclusively proprioceptive responses, neurons in areas 3b and 1 exhibit exclusively cutaneous ones, and neurons in area 2 receive convergent input from these two streams of somatosensory information (1). The cutaneous stream of somatosensory cortex is organized hierarchically, with area 3b at the bottom, area 2 at the top, and area 1 in the middle. The size of receptive fields increases and the response properties become increasingly complex at higher processing stages. On the proprioceptive side, area 3a is lower along the hierarchy than area 2. Somatosensory cortex sends projections to secondary somatosensory cortex and the parietal ventral area, located in the lateral aspect of parietal cortex, which contain high-level somatosensory representations (1). Neurons in these areas tend to exhibit tactile and proprioceptive responses, but the coding principles in these areas remain elusive, so their relevance to neuroprosthetics is uncertain.

10.3. Relative Merits of Different Somatosensory Targets

For individuals with tetraplegia, stimulation of the somatosensory nerves to convey sensory feedback—the preferred method for amputees—is not an option since the connection between nerve and brain is compromised. The most common target for electrical interfaces with the central nervous system to restore somatosensation is somatosensory cortex, which offers several advantages. First, area 1 is located principally or entirely on the crown of the postcentral gyrus, which makes it easily accessible with array technologies that are currently approved for human use. Second, the well-defined somatotopic organization of somatosensory cortex can be leveraged to convey information about targeted body parts, in particular the individual digits. Third, somatosensory cortex is a large structure, which, in principle at least, can be exploited to convey artificial feedback at a higher spatial resolution than the smaller upstream structures. The main disadvantage of somatosensory cortex as a locus for somatosensory restoration is that it is relatively high along the somatosensory hierarchy. Indeed, whereas the pathway sketched out above—called the medial lemniscal pathway—mediates the conscious experience of touch and proprioception, subcortical sensorimotor loops, for example spinal and cerebellar loops, play a key role in the online control of movement. Bypassing these loops may limit the efficacy of interfaces with somatosensory cortex to guide motor control.

The CN may also be a viable target for neural interfaces. As is the case with somatosensory cortex, the CN is organized somatotopically, although this somatopy is not as well mapped out as its cortical counterpart (266, 268). A potential advantage of the CN is that individual neurons carry more elementary signals about limb state, e.g., the length of a single muscle, and of contact events, deformations of a small patch of skin, than does cortex. In principle, these elementary feature detectors might be activated in combination to convey information about more complex stimulus features. The more elaborate response properties of cortical neurons may be more difficult to tap into. Another advantage is that the CN is the lowest structure along the somatosensory neuraxis that is still above any spinal cord injury (thus a viable target for tetraplegic patients), so interfaces with the CN bypass fewer sensorimotor loops than their cortical counterparts. However, a neural interface with the CN faces several major challenges. First, it is a very difficult structure to access surgically (266, 269). Second, it is very small (∼20 mm³ in humans), thereby offering only limited spatial resolution given the limits imposed by electrical stimulation (see below). Third, it is close to vital brain regions that, if damaged, can have severe consequences, including death.

A neural interface with the VPN—intermediate between CN and cortex—would also benefit from somatotopic organization (264, 265). A major advantage of a thalamic interface is that this structure is commonly accessed for deep brain stimulation, so the targeting approaches and implant technologies are already in place. Furthermore, VPN is not as vulnerable a surgical target as the CN. In amputees, microstimulation of VPN has been shown to evoke tactile sensations experienced on the missing limb (270–272). However, a VPN interface would also bypass important sensorimotor loops and is a relatively small structure compared to cortex (though larger than CN), so it is not clear that the advantages of a thalamic interface outweigh the drawbacks.

10.4. Effect of Electrical Stimulation on Neuronal Activity

Electrical stimulation involves the transfer of charge from the electrode tip, where the metal is exposed, to the tissue surrounding it, which in turn activates nearby neurons by polarizing their membranes (273). There are two types of processes involved: faradaic and capacitive. A faradaic process consists in converting a metal ion into an ion in electrolyte during the cathodic phase (reduction) and the reverse in the anodic phase (oxidation). A capacitive process involves rearranging ions around the electrode. Typically, stimulating electrodes are composed of noble metals (platinum or platinum/iridium) or iridium oxide because these do not corrode, as they might otherwise in the salt water solution of the brain.

Electrical stimuli typically consist of trains of square pulses that are charge balanced (anodal and cathodal currents are equal) to avoid an accumulation of charge in the neural tissue, which is liable to damage the tissue. Each electrical pulse evokes a volley of synchronous spikes in a volume of neurons around the electrode tip (274). Most spikes are initiated in axons, as these are more sensitive to electrical stimulation than the cell bodies and dendrites. At a first approximation, the probability of an electrical pulse evoking a spike in a given neuron is inversely proportional to the square of the distance between the electrode tip and that neuron’s nearest process to the electrode tip. That is, the current required to excite a neuron will increase with the square of the distance between the electrode and the neuron (FIGURE 9A) (274). Because the nearest axon to the electrode determines a neuron’s probability of activation, electrically activated cell bodies are diffusely distributed across the tissue, within a volume (275). Other factors also affect the electrical excitability of a neuron, including its morphology and myelination (276) as well as the conductivity of the tissue (277). Nonetheless, increases in the current amplitude will lead to recruitment of a greater neuronal volume, with the density of activated neurons decreasing progressively with distance from the electrode tip (278). The threshold current also depends on pulse duration, with longer pulses requiring less current, but levels off to a value called that neuron’s “rheobase current” (FIGURE 9B). The excitability of a neuron can be characterized by its chronaxie, which refers to the pulse duration that yields a threshold current of twice the rheobase current, typically between 100 and 400 µs.

FIGURE 9.

Biophysics of electrical stimulation of neuronal tissue. A: radial distance of direct activation of neurons, that is, the amount of current required to elicit a spike 50% of the time with a single cathodal pulse of 0.2-ms duration as a function of distance from the electrode tip. K, current-distance constant. B: amount of current at a given pulse duration required to elicit a spike 50% of the time with a single cathodal pulse. A chronaxie is the pulse duration at a threshold current level of twice the rheobase current (I_rh). Adapted from Ref. 274, with permission from Journal of Neurophysiology.

The spatial pattern of activation is dependent not just on the amplitude of ICMS but also on its frequency. Indeed, sensitivity to stimulation increases (thresholds decrease) as frequency increases up to ∼250 Hz (279), so a greater volume of neurons will be activated at higher frequencies than at low ones (280).

The volley of spikes induced by an electrical pulse is typically followed by a period of hyperpolarization that can last tens of milliseconds (281–283) and is also observed in downstream targets of the electrically stimulated neurons (282).

10.5. Safety

Direct-current ICMS can be used to produce lesions, which can then be used as landmarks to reconstruct the locations in the tissue where neuronal recordings were obtained. At sufficient intensities, even charge-balanced ICMS can damage the neuronal tissue. The intensity of an electrical stimulus can be expressed as the amplitude of each pulse (measured in amperes) and by its duration, both of which impact its total charge (in amperes per second or coulombs) (284). Another important consideration is charge density—measured in coulombs per unit area—which is most predictive of the damage the tissue might incur. ICMS at 20 µA—corresponding to 2 nC/phase for a pulse duration of 200 µs and 200 µC/cm² given the area of a typical electrode tip—applied daily for months causes neuronal death when the stimulation pulses are constant over minutes or hours (285). However, ICMS does not seem to have an impact on the tissue—up to 20 nC/phase—when it is interleaved with periods of no stimulation (286). Although the dependence of tissue damage on injected charge has not been systematically characterized, ICMS amplitudes rarely exceed 100 µA and pulse trains are typically limited in maximum duration to avoid damage as a precaution.

10.6. Perceptual Correlates of ICMS

The design parameters for ICMS include the intensity and frequency of pulse trains as well as the location on the cortical sheet where they are delivered. As might be expected, the evoked sensory experience varies substantially with the parameters of stimulation. In some cases, the relationship between ICMS parameter and some aspect of the evoked percept is straightforward. In many cases, however, that relationship is complex.

10.6.1. Location

Electrical stimulation of somatosensory cortex has long been known to evoke tactile sensations. Indeed, searching for epileptic foci, Wilder Penfield and colleagues (75) showed that stimulation of the surface of the postcentral gyrus evoked sensations that ranged from paresthesias to naturalistic tactile sensations. Famously, stimulation at different locations evoked sensations referred to different parts of the body, leading to the discovery of the somatosensory homunculus.

ICMS applied to somatosensory cortex, including areas 3b and 1, leads to highly localized tactile percepts (FIGURE 10). Indeed, somatotopic organization implies that all the neurons in a given volume of somatosensory cortex respond to a common patch of skin, i.e., these neurons send a common message about where a touch occurred. Their activation is thus interpreted by downstream structures as signaling the presence of a tactile stimulus at that skin location and, accordingly, gives rise to a sensation that is experienced there. The location of the evoked percept coincides with the location of the receptive fields of the neurons around the stimulating electrode (287, 288). For a given electrode, the location of its projected field—the patch of skin where an electrically evoked sensation is experienced—is highly stable over periods of months or years (289). The constancy of these projected fields is due to the remarkable stability of body maps in the somatosensory cortex of adults (290). That projection fields are highly localized and systematically mapped over the cortical sheet can be exploited to convey information about the location on the prosthesis of contact with objects. Whereas the relationship between the location of the electrode in the brain and the corresponding location of the projected field is systematic and stable, stimulation at different locations evokes percepts that also differ in other ways as well (see below).

FIGURE 10.

Implantation of electrode arrays in somatosensory cortex. A: diagram of the palmar surface of the hand of a subject with 2 Utah arrays implanted in somatosensory cortex. The shaded regions correspond to the locations of electrically evoked sensations. B: projected fields progressed systematically over the electrode arrays, as expected from the somatosensory homunculus. C: location of the arrays in motor and somatosensory cortex. Arrays to right of the central sulcus (posterior) are the somatosensory ones. Colors denote regions of activation during attempted digit movements, identified through magnetoencephalography (MEG). Reproduced from Ref. 289, with permission from Science Translational Medicine.

10.6.2. Intensity

Increasing the current delivered to the brain, by increasing the amplitude of the pulse or its width, leads to an increase in the magnitude of the evoked percept, a relationship that is approximately linear (FIGURE 11A) (289), as long as the stimulus is sufficiently strong to evoke a sensation. Indeed, pulse trains that fall below a certain amplitude—the detection (or absolute) threshold—are too faint to be reliably perceived. The detection threshold can be measured by having a subject report in which of two sequential intervals an electrical pulse train was presented (chance performance is 50%). Stimulus amplitudes are chosen such that some are expected to be perceived and others not. The detection threshold is defined as the stimulus amplitude that yields 75% performance on the task. Detection thresholds vary widely from electrode to electrode, ranging from ∼15 µA up to 50 µA or more (279, 289, 291). As alluded to above, detection thresholds decrease with frequency and level off at ∼250 Hz.

FIGURE 11.

Assessing sensitivity to electrical stimulation. A: perceived magnitude. reported by a human subject, increases linearly with stimulation amplitude. Reproduced from Ref. 289, with permission from Science Translational Medicine. B: in monkeys. the detection threshold, defined as the current amplitude that yields 75% correct detection performance, is on average ∼30 µA but can be as low as 15 µA. Reproduced from Ref. 291, with permission from IEEE Transactions on Neural Systems and Rehabilitation Engineering. C: the just-noticeable difference (JND), defined as the change in current amplitude that is correctly discriminated 75% of the time, is on average ∼30 µA in monkeys and can be lower in humans. The JND remains constant across reference amplitudes, thereby violating Weber’s law. Reproduced from Ref. 279, with permission from Proceedings of the National Academy of Sciences of the United States of America.

Another way to gauge sensitivity to electrical stimulation is by assessing subjects’ ability to discern changes in stimulus amplitude. In a typical assessment, subjects are sequentially presented with a pair of pulse trains and report which of the two was more intense. The just-noticeable difference (JND)—the change in stimulus amplitude that yields 75% performance on this task—can be used as a gauge of sensitivity to amplitude changes. JNDs are on average ∼30 µA in monkeys (279), whereas humans are more sensitive, yielding JNDs on the order of 15 µA (289). As is the case with detection thresholds, JNDs vary widely across electrodes. With natural stimulation, JNDs increase in proportion to the reference stimulus, a phenomenon known as Weber’s law (292). With ICMS, amplitude JNDs do not scale with the reference amplitude, suggesting that the variability in ICMS-evoked neuronal responses does not increase in proportion to the mean strength of the responses as is observed with natural stimulation (279). The detectable and safe range of stimulation—between 15 µA (the detection threshold) and 100 µA (the maximum safe amplitude)—thus only allows for a few discriminable increments, between three and six depending on the electrode. Given the relatively coarse resolution at which changes in amplitude can be perceived, one might hope to achieve better sensitivity by stimulating through multiple electrodes at the same time. However, sensitivity to stimulation delivered synchronously through four electrodes—where each electrode is stimulated at the same level—is equivalent to that when the most sensitive of the electrodes is stimulated alone, even though the latter approach only delivers a quarter of the current (Ref. 293, but see Ref. 294).

Increases in current amplitude have an effect on the local neuronal population that is broadly analogous to increases in pressure (288, 295). Indeed, applying greater force results in increased neuronal activation and in the recruitment of neurons. Modulation of ICMS amplitude can thus be used to convey information about applied pressure (see below). One might expect that higher currents—which activate larger volumes of cortical neurons whose receptive fields radiate outward—would evoke sensations that are experienced over larger swaths of skin. However, the reported area over which an electrically induced tactile sensation is experienced does not change systematically with changes in current amplitude (296). Note, however, that increases in the pressure applied via a probe of fixed size also lead to recruitment of additional neurons in somatosensory cortex without concomitant changes in the perceived extent of the stimulus (295). The ability to discern changes from intensity from changes in extent likely stems from differences in the spatial pattern of neuronal activation evoked by broad, low-intensity skin indentations and small, high-intensity ones. Changes in the spatial profile of activation induced by electrical stimulation presumably mimic changes induced by changes in pressure rather than by changes in the spatial extent of a skin indentation.

10.6.3. Frequency

As mentioned above, sensitivity to ICMS, as gauged by the detection threshold, increases with pulse frequency up to ∼250 Hz, at which point it levels off (279). One might thus expect that increases in frequency lead to increases in perceived magnitude, as is the case with electrical stimulation of the nerve (297). Although this is the case in monkeys (298), the same may not be true in humans (299). However, changes in ICMS frequency also have an impact on the quality of the sensory percept. Quality refers to aspects of the sensory experience beyond its intensity, spatial extent, or duration. For example, a touch can be described as a texture, as an indented edge, or as a moving surface, all of these descriptors indicating different qualities. Indeed, animals are able to distinguish changes in frequency independently of changes in amplitude, at least on some electrodes, up to ∼200 Hz. As mentioned above, each ICMS pulse evokes a spike in a large population of neurons. A periodic pulse train will thus elicit a phase-locked response in the activated population at the stimulation frequency. Accordingly, the population firing rate increases approximately linearly with frequency but is also dependent on amplitude. Monkeys’ ability to distinguish frequency independently of amplitude suggests that their behavior does not depend solely on the strength of the evoked response at the population level. Rather, animals may rely on the temporal patterning in the neuronal response (298). Changes in ICMS frequency have also been shown to impact sensory quality in studies with human participants (299, 300). Low-frequency ICMS tends to evoke intermittent tapping or pressure sensations, whereas high-frequency stimulation tends to evoke tingling or vibratory sensations.

10.6.4. Heterogeneity of the sensory experience

As discussed above, stimulating through electrodes at different locations evokes sensations that are experienced on different parts of the body, in a way that is predicted by the body maps in somatosensory cortex (288, 289). The sensory experience evoked through stimulation of different electrodes also varies in other ways. First, the detection thresholds and amplitude JNDs vary widely across electrodes (279, 288, 289). Second, an animal’s ability to distinguish ICMS frequency independently of amplitude varies across electrodes: On some electrodes, animals can distinguish changes in ICMS frequency nearly independently of changes in amplitude, whereas on others changes in frequency or changes in amplitude seem to have indistinguishable sensory correlates (298). Third, the quality of the sensation evoked by a given ICMS train varies widely: Some electrodes are more liable to produce tapping or pressure sensations, others tend to produce vibratory or tingling sensations, and still others produce tactile motion percepts (289, 299–301). Remarkably, sensations are described as being natural or nearly so; that is, humans experience ICMS-evoked tactile sensations as being similar to sensations that might be experienced through natural touch. This is surprising given the unnatural patterns of activation that are evoked via ICMS: each pulse synchronously activates neurons within a volume of tissue, regardless of their coding properties. Synchronized, periodic activation of large populations of neurons is not observed in healthy cortex and yet evokes nearly natural sensations.

Differences in the properties of the sensory experience—overall sensitivity, quality, etc.—seem to be unrelated to the electrodes themselves (e.g., impedance, metallization). One possibility is that across-electrode differences in the sensory correlates of ICMS are due to differences in the depth of the electrode tip within the cortical tissue. Indeed, detection thresholds have been shown to be lower (and thus sensitivity higher) when the electrode tip is deeper in the tissue (302). Depth effects might thus account for across-electrode differences in detection thresholds, because, even with a planar array, electrode tips may not end up at the same depth within the tissue. However, sensitivity varies more across electrodes than would be expected given the dependence of sensitivity on depth. Other differences in the sensory experience across stimulation sites, such as differences in sensory quality, are unlikely to be mediated by differences in the electrodes or in the depth of their tips. Indeed, suprathreshold stimulation directly activates multiple cortical layers, thereby reducing the impact of depth.

Rather, variability in the sensory experience likely reflects differences in the function of activated neurons. A case in point is the relationship between ICMS frequency discrimination and submodality input. In area 3b, the submodality input of neurons around the stimulating electrode—that is, the subpopulation of nerve fibers from which they receive their dominant input—is predictive of animals’ ability to distinguish changes in frequency of ICMS delivered through that electrode (303, 304). The submodality input of a neuron is one of the factors that determine its function. This relationship between submodality input and ICMS-evoked percepts is consistent with the view that activation of neurons with different functions will have different sensory consequences.

The dependence of the electrically evoked sensation on locus of stimulation hinges on two aspects of cortical functional organization. First, somatosensory cortex is not a homogeneous neural structure: Cortical neurons exhibit a wide variety of response properties: Some neurons are well suited to encode surface texture, others tactile motion, etc. (1). Second, neurons with similar response properties tend to cluster together. Neurons that are activated by an ICMS pulse train are thus more likely than not to have some overlap in their function, and the evoked sensory experience will reflect this. For example, one patch of cortex may contain a greater preponderance of neurons that play a role in fine texture perception. Electrical activation of these neurons might thus be liable to evoke a texture-like percept. Activation of an adjacent population of neurons implicated in motion processing might evoke a motion percept (300). The patchy distribution of sensations is consistent with a view that the functional properties of neurons are not homogeneous across the cortex. However, the relationship between the functional properties of a neuronal population and the quality of the sensory percept evoked when that population is electrically activated has not been investigated. The hindrance is that such questions about sensory quality can only be addressed in a human subject, endowed with language, and human subjects enrolled in BMI experiments tend to be at least partially deafferented, so the native coding properties of neurons in somatosensory cortex cannot be characterized. Nonetheless, the mapping between sensory quality and location may be revealed by having subjects imagine different qualities while monitoring responses over the cortical surface (301).

10.7. The Problem of Proprioception

Like touch, proprioception is critical to upper limb function as evidenced by the severe deficits incurred when proprioception is abolished (305). The importance of proprioception extends to BMIs: a human participant was tasked with moving a brain-controlled bionic hand back and forth across a line positioned approximately at her midline. She rapidly lost track of her arm, and thus her ability to perform the task, when vision was obscured (306). Even in the context of manual control, proprioceptive signals convey information about the state of the hand (261, 307), which in turn is critical to sensing the structure of grasped objects (1, 308). In light of this, efforts have been underway to evoke proprioceptive sensations through ICMS of a cortical field that has been implicated in proprioception, namely Brodmann’s area 2. Indeed, neurons in area 2 exhibit proprioceptive responses and contain a representation of the proximal limb and hand (307, 309–311). Furthermore, the responses of neurons in area 2 depend on the direction in which the limb is moving, much like their counterparts in M1 (309). This representation of movement could in principle be leveraged to convey feedback about limb state. However, experiments with monkeys suggest that electrical stimulation of neurons with a given preferred direction does not evoke a sensation of limb movement in that direction (312). Furthermore, human participants rarely report sensations of limb movement through electrical stimulation of somatosensory cortex. Indeed, sensations reported as proprioceptive are often described as movement of the skin and not around joints as is the case with (natural) proprioception (300). One possibility is that proprioceptive sensations have not been elicited through electrical stimulation of the postcentral gyrus—where human-approved technology can be deployed—because area 2 is not strictly proprioceptive but rather is involved in integrating tactile and proprioceptive signals, putatively to compute the location of contacts in three-dimensional space. A potential avenue toward artificial proprioception would be to stimulate Brodmann’s area 3a, in which neurons are exclusively driven by joint movements (1, 261). Area 3a stimulation has been shown to evoke discriminable percepts in monkeys (313), but the quality of these percepts and its dependence on stimulation parameters is unknown.

10.8. Closed-Loop Control

10.8.1. ICMS can be used to guide behavior

Ultimately, ICMS-based feedback needs to be applied in the context of behavior. In early studies of ICMS-based feedback, sensory signals were used to guide behavior in ways that are approximately akin to using touch to recognize objects in the dark. In one case, monkeys explored virtual “textures” with a brain-controlled arm avatar, searching for a target texture (one of two or three; FIGURE 12A) (314). Positioning the hand within experimenter-defined zones in the workspace led to the delivery of temporally patterned pulse trains, with different zones leading to different patterns. The location of the zones changed from trial to trial, and the animal explored the space to find each zone with the goal of identifying the target zone, corresponding to a specific pattern of ICMS (314). Animals were able to quickly learn to use the tactile feedback to perform the task. In one variant of this task, a human subject searched for a specific “object” in a “handbag,” signaled by ICMS (315).

FIGURE 12.

Using intracortical microstimulation (ICMS) to guide behavior. A, left: an arm avatar was controlled either with a joystick or through brain signals. When the hand hovered over a virtual object, an ICMS stimulus was delivered to somatosensory cortex. The animal was rewarded if it lingered on the target object. Right: the animals learned to explore the workspace and find the target with hand control (HC) and ultimately brain control [brain control with hand movements (BCWH)]. Reproduced from Ref. 314, with permission from Nature. B, left: overhead view of the Action Research Arm Test (ARAT) showing the object presentation position (green dot) and the raised platform target (green box). Different objects (not all objects shown) were positioned at a standard location, grasped, and then placed on the platform as quickly as possible. For all tasks, the arm was under full control of the user from the start to the end of a trial. Center: comparison of ARAT scores before experiment onset, which spanned a range of controlled degrees of freedom and occasionally employed ICMS feedback (blue dots), to scores with ICMS feedback (blue) and without (gray). ARAT scores with ICMS feedback were significantly higher than scores without. Right: empirical cumulative distribution of individual trial times, including failed trials, shown on a log-normalized axis. Vertical red dashed lines indicate when 50% of successful trials were completed. Data to left of the vertical green dashed line represent trials completed in <5 s. Shading indicates the 95% confidence bounds, calculated with Greenwood’s formula. Reproduced from Ref. 6, with permission from Science.

ICMS of somatosensory cortex has also been used for navigation. In one study, rats were trained to find an invisible target by following ICMS signals (316). Specifically, the strength of ICMS was modulated to the extent that a photosensor fixed to the rat’s head was pointed toward the target, which emitted invisible light. After repeated exposure, the animals learned to use ICMS signals to find the targets. In a similar study, monkeys learned to reach toward unseen targets, with direction and proximity information conveyed through ICMS (317), a task that has also been applied to a human participant (318). Finally, surface stimulation of somatosensory cortex in a human subject was used to guide the aperture width of an intact hand (319), a one-dimensional analog to the two-dimensional navigation tasks described above.

10.8.2. Functional benefits of artificial touch

Although the studies summarized above provided proof of principle that ICMS-evoked sensations can be used to guide behavior, the artificial touch was used to explore a workspace rather than guide object manipulation, the more ethological use of touch. Indeed, the tasks would have been trivial had visual information been available, as is object recognition or navigation with vision as opposed to without. During object interactions, tactile feedback provides information about when the hand makes contact with objects, where on the hand the object makes contact, and the dynamics of the applied force at each contact point (3). Basic information about object interactions—contact timing, location, and force—is poorly conveyed visually. First, contact points between hand and object are often occluded. Second, vision conveys little to no information about the force applied on all but the most compliant objects. Third, visual signals are much slower than their tactile counterparts in signaling when contact with an object is established.

To intuitively convey information about contact events, the output of sensors located on a prosthetic hand can drive ICMS through electrodes with somatotopically appropriate projection fields. For example, the index fingertip sensor on the prosthesis drives electrical stimulation of an electrode with a projection field on the index fingertip. That way, anytime the user makes contact with an object with the (prosthetic) index fingertip, for example, he or she will experience a sensation on the index fingertip, thereby intuitively signaling contact location. The timing of the sensory experience will correspond to the timing of the contact, thereby conveying rapid feedback that contact is established. Finally, the time-varying force can be conveyed by modulating the ICMS amplitude—greater force resulting in stronger ICMS—which in turn modulates the magnitude of the resulting percept in a way that is broadly analogous to the sensory correlates of changes in applied force (288, 295).

The benefit of artificial touch for object manipulation with a brain-controlled bionic hand was tested with a standard clinical test of limb function, used to assess the sensorimotor sequelae of stroke: the Arm Research Action Test (ARAT) (FIGURE 12B). The ARAT consists of having the participant grasp and transport objects from one location to another. The main index of performance is the speed with which objects are transported (or the number of objects transported over a fixed period of time). Performance on this task was significantly improved with artificial touch from without, despite the constant presence of visual feedback (6). The participant reported that a main benefit of the artificial touch was the rapid feedback about contact timing. Accordingly, he was more confident in reaching rapidly for the object, knowing that he would receive immediate feedback as soon as he touched it. Indeed, ICMS-evoked percepts arise systematically faster than visual ones (320).

All of the objects presented in the ARAT were solid objects, so modulating forces was not necessary, as it would be to manipulate fragile objects, for example (255, 257). Our ability to exert just enough force on an object to it up and manipulate it without dropping it depends on the sense of touch. The manual forces exerted by an insensate bionic hand are likely to be much higher than are those by a sensate one, but this has not been investigated.

10.8.3. The problem of electrical artifacts

Closed-loop control involves reading out control signals from brain regions involved in motor control and writing in feedback signals to brain regions involved in perception. This bidirectional communication presents an additional challenge, namely that ICMS disrupts the ability to read out neuronal signals. Indeed, an individual pulse involves voltage transients measured in volts whereas neuronal signals are measured in microvolts, six orders of magnitude smaller. Each ICMS pulse thus drowns out any neuronal signal over a period of time, even if recording and stimulation sites are centimeters apart. Even stimulation at the level of muscles in functional electrical stimulation applications can produce large electrical artifacts on intracortical microelectrode recordings (321). The problem of the electrical artifact can be mitigated by using special amplifiers that allow charge to dissipate rapidly and by using signal processing to remove components of the potential that can be attributed to the stimulating pulse to isolate spike-related potentials (322). Regardless, hardware and signal processing strategies are not sufficient to completely eliminate the stimulation artifact, and signals obtained during each stimulation pulse and shortly thereafter are discarded. Accordingly, high-frequency ICMS is undesirable in the context of closed-loop control because the more frequent pulses require discarding more of the control signals.

10.9. Next-Generation Artificial Touch

As sketched out above, current sensory encoding algorithms are inspired by known coding properties of the somatosensory system. For example, mapping between bionic sensor and electrode leverages the body maps in somatosensory cortex, and pressure feedback is inspired by the measured effects of changes in pressure on the evoked neuronal response. These algorithms convey sufficient information about contact events to improve the dexterity of brain-controlled bionic hands, even in the presence of vision (6). The next generation of algorithms will incorporate other known aspects of tactile coding in cortex.

For example, current algorithms track the pressure exerted on the object using a linear mapping between sensor output and ICMS amplitude. However, the response of populations of neurons during the initiation, maintenance, and termination of contact is characterized by massive transient responses—an increase in the rate of response and in the size of the activated population—at the onset and offset of contact (295) (FIGURE 13). These transient responses are >10 times stronger than are their counterparts during maintained contact. This phenomenon reflects the fact that neurons are more sensitive to changes in skin deformation than they are to static skin deformations. This property of the aggregate neuronal response in cortex is inherited from the periphery, where the same phenomenon is observed. Sensory encoding algorithms for peripheral nerve interfaces that mimic this sensitivity to contact transients (323) have been shown to confer greater dexterity to myoelectrically controlled bionic hands than do the standard pressure tracking algorithms (255, 257). Given the similarity of the peripheral and cortical responses at the aggregate level (295), the same principles are likely to yield improvements for brain-controlled prosthetic hands endowed with ICMS-based feedback.

FIGURE 13.

Spatiotemporal dynamics of cutaneous responses in somatosensory cortex. A: response to 70 repeated presentations of an indentation delivered to the little finger, the profile of which is superimposed on the raster. B: peristimulus time histogram of the response at the hot zone electrode to the same stimulus. Shaded area denotes the SE. The leftmost edge of the time scale bar indicates stimulus onset. C: dynamics of the spatial extent of cortical activation as a function of time. Adapted from Ref. 295, with permission from Cerebral Cortex.

As discussed above, the quality of sensation is electrode dependent, a phenomenon that likely reflects differences in the coding properties of local neurons. One of the functions of nervous systems is to extract behaviorally relevant information from peripheral sensory input. Different populations of neurons in somatosensory cortex explicitly encode certain types of sensory information that are not encoded by upstream neurons (1). For example, a subpopulation of cortical neurons is selectively sensitive to tactile motion (324, 325). Sensory encoding algorithms may be able to harness these feature maps to convey information about high-level object features. For example, object slip might be signaled by activating populations of motion-sensitive neurons. Indeed, the visual system provides a proof of concept: activation of motion-sensitive neurons in medial temporal cortex—an area of the brain specialized for the processing of visual motion—evokes a visual motion percept (326). The ability to evoke visual sensations of directed movement is predicated upon the functional topographies in medial temporal cortex: neurons with similar direction preferences are clustered together. To harness feature selectivity in somatosensory cortex for use in artificial touch will require that touch representations also be topographically organized, which remains to be seen. That stimulation through different electrodes evokes sensations with different qualities may be interpreted as evidence for functional organization, but this phenomenon has yet to be harnessed in the context of a BMI.

Interactions with objects typically involve multiple contact points distributed over the entire hand (327). Each contact results in a spatio-temporal activity spanning multiple millimeters of cortical surface (295). However, most work on artificial touch has focused on single electrodes, involving activation of local populations spanning less than a millimeter of cortical surface: 100 µA, the maximum confirmed safe current, activates neurons ∼0.3 mm from the electrode tip (FIGURE 9) (328). Stimulating through multiple adjacent electrodes may thus allow for the elicitation of more naturalistic patterns of neuronal activation. In addition, little is known about interactions across electrodes in the evoked perceptual experience. The assumption is that the sensory experience evoked by ICMS through one electrode is independent of stimulation at other electrodes, even though there is anecdotal evidence to the contrary. The exploration of multichannel ICMS opens up a world of possibilities to improve the naturalness and intuitiveness of the evoked percepts (329).

11. ALTERNATE SIGNAL SOURCES FOR EFFERENT BMIs

In this review, we focus on intracortical BMIs, where penetrating electrodes provide high-resolution readouts of the activity of individual neurons, small groups of neurons, and local field potentials. However, other methods to monitor brain activity have been tested for BMIs or related applications (FIGURE 14). Although a comprehensive review of all neural interfaces is beyond the scope of this article, we explore the trade-offs inherent to the different approaches to examine their potential within the BMI space.

FIGURE 14.

Neural interface alternatives. A: electroencephalographs (EEGs) use large electrodes to record field potentials from the surface of the scalp. Epidural electrocorticography (Epi) records epidural field potentials (EFPs) from the surface of the dura. Subdural electrocorticography (ECoG) records directly from the surface of the brain; intracortical electrodes penetrate the brain. Adapted from Ref. 332, with permission from Neuroscientist. B, left: electrical signals (blue) are distorted during transmission from the brain to the scalp surface, whereas magnetic fields (purple) are not. MEG, magnetoencephalography. Adapted from Ref. 439, with permission from TKK Dissertations. Right: functional near-infrared spectroscopy (fNIRS) uses a paired emitter and detector to shine near-infrared light onto brain tissue and measure the hemodynamic response. Adapted from Ref. 351, with permission from Journal of Biomedical Optics. C: emerging endovascular interfaces tunnel through vasculature to place electrodes within cerebral veins without requiring a craniotomy. Adapted from Ref. 396, with permission from Journal of NeuroInterventional Surgery.

11.1. Noninvasive Interfaces

11.1.1. Electroencephalography

Perhaps the most widely used noninvasive brain interfacing method is the scalp electroencephalogram (EEG), one of the oldest methods for measuring brain activity (FIGURE 14A). Scalp EEG provides a coarse readout of brain activity: potentials on a single electrode may reflect electrical activity integrated over 10 cm² or more, due to volume conduction between the brain and scalp (330, 331). Scalp EEG thus captures the activity of tens of millions of neurons acting in concert (331) and constitutes a spatially and temporally smoothed version of the LFP signals recorded from intracranial electrodes (330).

The use of EEG signals for BMIs can be classified into two categories of approaches (332). In one case, a sensory stimulus—typically visual—is presented and the evoked response is monitored via EEG. The user can then volitionally modulate certain features of the response to each stimulus, for example the so-called P300, a spike in neuronal activity ∼300 ms after stimulus onset (333–335), or the steady-state response to an ongoing stimulus (336–338). The resulting binary signal can be then used to control switches or to type by selecting a letter when it is highlighted on the display. This approach is not applicable to limb control.

Another strategy consists of leveraging EEG signals for continuous control of cursors and, in principle at least, bionic limbs. Typically, the EEG signal is filtered to isolate the mu (8–12 Hz) or beta (12–25 Hz) bands in sensorimotor cortical areas, which undergo a desynchronization when a subject makes or imagines a limb movement (339). Users can then learn to modulate these signals over the course of several sessions (340, 341). However, because of the limited spatial resolution of the signal, imagined movement of a given limb results in a stereotyped pattern of desynchronization across a broad area of cortex, obscuring information about individual limb segments. Accordingly, to control multiple degrees of freedom independently, users must learn nonintuitive strategies, such as combining imagined movements of both hands and both feet (339).

Several practical challenges also limit the utility of scalp EEG for long-term assistive devices for people with movement impairments. Measurements require donning and doffing a cap and applying gel for higher-quality signals. In surveys of potential users with spinal cord injuries, the visibility of such technologies and the potential impact on cosmesis and convenience are key factors that limit enthusiasm (14). An additional challenge is that ∼20% of subjects are unable to attain control of EEG-based BMIs, potentially because of variations in brain structure that cause the relevant control signals to be undetectable from the scalp, a problem known as “BCI illiteracy” (342–344).

11.1.2. Magnetoencephalography

Like EEG, magnetoencephalography (MEG) involves recording signals that reflect the large-scale concerted activity of neurons. When measured noninvasively, magnetic signals have some advantages over electrical signals. In particular, electrical signals are influenced by currents in the interstice between the brain and scalp and distorted by various inhomogeneities in the head, whereas magnetic signals are not (345) (FIGURE 14B). The improved signal transmission confers higher spatial resolution to MEG than scalp EEG. Accordingly, MEG signals exhibit single-trial correlation with intended movements (346–348). However, the magnetic inductions produced by intracranial activity are quite small—typically 7–8 orders of magnitude smaller than the Earth’s magnetic field—and thus require extremely sensitive magnetometers to detect (349), typically accomplished by large, liquid helium-cooled superconducting interference devices (SQUIDs). MEG signals are also extremely sensitive to electronic artifacts, so measurements are generally made in dedicated rooms with multilayered magnetic shielding and without any metal moving nearby. Thus, MEG requires capital investment on the scale of MRI systems (349) and has been confined to restricted physical spaces, making it largely impractical for assistive BMI applications. In recent years, an alternative MEG technology based on optically pumped magnetometers has demonstrated sensitivity on par with SQUID systems without the need for cryo-cooling (350). This new technology may lower costs substantially and also enable wearable devices, which has sparked commercial interest from companies such as Kernel. Although the technology may be game-changing for applications in restricted physical settings, a general-purpose and mobile assistive BMI would still require new solutions to mitigate the influence of electronic artifacts.

11.1.3. Functional near-infrared spectroscopy

Neuronal activity in the brain results in a hemodynamic response, i.e., changes in blood flow, volume, and oxygenation. Oxygenation, in particular, modulates the absorption properties of brain tissue. Because near-infrared light is minimally absorbed or scattered as it travels through the scalp and skull, an emitter and a detector at the scalp (typically at a distance of 3–4 cm) can be used to shine near-infrared light onto brain tissue and measure reflected light to noninvasively detect changes in absorption due to the hemodynamic response (FIGURE 14B). By increasing the distance between the emitter and detector, deeper tissue can be imaged. Functional near-infrared spectroscopy (fNIRS) thus provides another noninvasive modality to monitor brain activity that may be germane to BMIs (351–353). However, fNIRS suffers from two critical weaknesses that limit its potential. One is the slow timescale of the hemodynamic response, as vascular changes occur several seconds after the associated neural activity (351, 352), yielding an information transfer from fNIRS-based BMIs that does not exceed 4 bits/min (352), much lower than transfer rates from other interfaces typically measured in bits per second (38). Second is the coarse spatial resolution—between 1 and 3 cm (354)—that precludes simultaneous control of multiple degrees of freedom. The application of fNIRS to BMI has recently been the subject of some controversy after a demonstration of fNIRS-based communication in subjects who were completely locked-in because of advanced ALS (355). A reanalysis of the collected data failed to replicate the findings and led to retraction of the original paper (356, 357).

11.1.4. Electromyography

Myoelectric control is a common strategy for prosthetic arm control by amputees (358). For people with spinal cord injury, muscle activity above the site of lesion can be harnessed as a control signal for FES-based prostheses (359). Perhaps surprisingly, individuals diagnosed with “complete” spinal cord injury exhibited volitional control of residual muscle activity below the site of injury (360, 361). The preserved connections may prove useful for prosthetic control even if they cannot support functional use of the muscles. For example, activity can be leveraged through remapping strategies, e.g., by using toe activity as a control signal for FES-based restoration of grasp (362). Additionally, recent work with paralyzed individuals has demonstrated that noninvasive, high-density electromyography (EMG) arrays can be used to record signals directly from forearm muscles of interest—such as individuated finger movements and wrist movements—and can even achieve single-motor unit resolution (363, 364). At present, this emerging approach has only been demonstrated in coarse movement classification tasks, and the fidelity and flexibility of the approach in comparison to other interfaces are unknown.

11.2. Semi- and Minimally Invasive Interfaces

11.2.1. Electrocorticography

Subdural electrocorticography (ECoG) is a widely used clinical approach for monitoring brain activity in patients with medically intractable epilepsy. In these patients, ECoG is used to delineate the epileptic seizure focus so it can be surgically resected and to map cortical fields associated with hand and face movements and language to avoid resecting them (365, 366). Electrode contacts are placed directly onto the pia mater above cortex and are often in place for more than a week as patients remain in the hospital (FIGURE 14A). ECoG provides access to large-scale sensorimotor rhythms below 50 Hz, which can also be monitored via EEG (including mu and beta oscillations), as well as more focal, broadband changes that occur at frequencies above 50 Hz and are largely inaccessible extracranially (365).

Of the neural interfacing alternatives, ECoG is most likely to provide performance comparable to intracortical devices. To date, the most widely tested ECoG devices—standard clinical ECoG grids—offer coarse coverage, with large contacts (>2 mm in diameter) spaced ∼10 mm apart, where each contact monitors LFP activity from >500,000 neurons (365). However, more closely spaced contacts, e.g., 1- to 3-mm spacing, allow for more focal monitoring of neuronal activity and have enabled higher-quality decoding of arm and hand movements and speech (367, 368). ECoG arrays with electrodes spaced even closer together (on submillimeter scale) may yield readouts at an even higher resolution (369–371). A potential advantage of ECoG is the ability to cover larger extents of the cortical surface than traditional intracortical devices [see, e.g., Ref. 370, though recent intracortical alternatives may minimize this difference (Refs. 372, 373)]. In general, neurons within a given area often provide redundant information, so achieving more diverse cortical coverage may prove highly advantageous for high-dimensional control. Additionally, the widespread deployment of ECoG-based responsive neurostimulation systems for epilepsy (374) provides a demonstration of long-term safety that could facilitate its clinical translation in the context of BMIs.

ECoG signals are often considered more stable than their intracortical counterparts. Indeed, for chronic implants with 2- to 3-mm contacts and 4-mm intercontact spacing, ECoG BMIs with fixed decoders have demonstrate relatively stable offline decoding performance (375) and can provide a substrate for subjects to achieve stable online to control over the course of months (10). Additionally, histological analysis after a chronic ECoG implant revealed minimal damage to the cortical tissue under the implant, although the device itself was encapsulated in collagenous tissue (376).

Despite the high signal stability (10, 377), standard ECoG devices yield far poorer performance than intracortical interfaces. For example, two-dimensional (2-D) cursor control for point-to-point movement tasks was worse with ECoG than with intracortical arrays (see TABLE 1). Similarly, although an ECoG participant improved substantially over the course of months on a grid task and achieved a bit rate of 0.79 bits/s (10), performance on this task with an intracortical BMI is markedly higher (TABLE 2). For example, one participant achieved 1.87 bits/s on his first-ever neural control session (173), whereas three others maintained high quality control for months to years after implantation, with bitrates ranging from 1.4 to 4.2 bits/s (38, 242).

Some evidence suggests that recently developed high-density ECoG devices with submillimeter resolution may improve performance, having shown recording longevity on the scale of months to years (378, 379). As such high-density ECoG devices become more widely tested, a key design parameter in maintaining signal stability may be minimizing tissue responses or fluid buildup (376, 380–382). ECoG interfaces have recently been used for control of powered exoskeletons, for both restoration and rehabilitation applications (383, 384).

Epidural field potentials (EFPs) are a variant of ECoG in which electrodes are placed on the outer surface of the dura mater (FIGURE 14A) (383, 385–387) to decrease the risk of bacterial encephalitis by isolating the implant from the intracranial space. Quantitative comparisons suggest that EFPs yield a decreased signal-to-noise ratio and spatial resolution relative to subdural ECoG (385) and thus provide a different balance between signal quality and invasiveness than ECoG and intracortical electrodes.

11.2.2. Stereoelectroencephalography

Stereoelectroencephalography (SEEG) provides an intriguing option for BMIs that has received little experimental attention (388–390). Like ECoG, SEEG is an invasive brain interface used to monitor brain activity in patients with pharmacologically intractable epilepsy. SEEG uses penetrating depth electrodes that allow access to deep structures, out of reach for intracortical or ECoG arrays (388). Implanting SEEG electrodes only requires drilling small burr holes into the skull, thereby reducing the risk of complications in comparison with the full-blown craniotomy required for intracortical electrode arrays. Given the well-established safety history for temporary (multiweek) SEEG implants (391–393), these might be well suited for applications that do not require permanent implants, such as neurorehabilitation. The depth electrodes are also well suited for applications that target deeper structures, such as the motor regions of the cingulate cortex. Whether SEEG takes hold in the BMI space remains to be seen.

11.2.3. Endovascular interfaces

The emerging field of endovascular interfaces is providing an alternate approach to monitoring electrical activity with reasonable fidelity, without requiring a craniotomy (FIGURE 14C). These interfaces tunnel through cerebral veins, which are split into those that lie externally on the cortical surface and internal veins that may even sit within sulcal folds. Transient endovascular recording of brain activity was initially reported in 1969 (394). Chronic endovascular recording devices have recently yielded signal quality comparable to that of epidural field potentials (395), thereby supporting restoration of some communication abilities in people with ALS (396).

11.2.4. Functional ultrasound

Functional ultrasound (fUS) is an emerging alternative to monitor brain activity (397–401). Like fNIRS and fMRI, fUS measures hemodynamic changes by monitoring changes in cerebral blood volume. To image these changes with high resolution, a craniotomy is performed and a small piezo-electric array is placed on the surface of the dura mater. High-frequency ultrasonic waves (∼15 MHz transmit frequency, imaged at ∼10 kHz) are transmitted into the brain by the array. Measuring the waves reflected at tissue interfaces reveals changes in cerebral blood volume. Such changes can be localized to regions on the order of 100 µm and are strongly correlated with slow fluctuations in neural firing rates (<0.3 Hz) at the imaged location (400). In a recent study with monkeys, fUS enabled single-trial decoding of upcoming movement directions during a delayed saccade and reaching task. Although the task was simple (two alternatives) and decoding was performed on large time windows (seconds), the study provides a proof of concept that fUS can extract single-trial information about movement intention (401). Further advances such as ultrafast ultrasound localization microscopy (402) hold promise for high-resolution noninvasive fUS.

12. NONINVASIVE SENSORY FEEDBACK

The development of noninvasive strategies to harness control signals from the brain entails parallel efforts to convey somatosensory signals noninvasively, typically by electrically or mechanically stimulating a sensate region of skin (403). As in a cortical interface, the output of sensors on the bionic hand drives the electrical or mechanical stimulus delivered to the skin. For example, the output on each fingertip sensor might drive the amplitude of each of five vibrators arranged on the neck. That way, thumb forces would result in vibratory sensations experienced at one neck location, index forces with sensations on another location, and so on.

12.1. Feedback Modalities

For both electrical and mechanical stimulation, information about contact events can be conveyed by stimulating through different contacts (as in the example above) or by modulating the stimulation parameters. For example, contact force can be tracked in a graded fashion by modulating the frequency or amplitude of stimulation (404–408). In some cases, contact information can be conveyed via more complex spatio-temporal mappings (409).

The advantage of mechanical stimulation is that it evokes naturalistic tactile sensations, in contrast to electrocutaneous stimulation, which typically evokes parasthetic sensations often described as unpleasant (410–412). On the other hand, mechanical stimulators are bulky, difficult to don and doff, and power hungry, requiring frequent recharging. As discussed below, one of the shortcomings of noninvasive approaches is their limited bandwidth. One way to overcome this limitation is to combine electrical and mechanical modalities, as these can be perceived semi-independently, even when delivered simultaneously to the same skin location (413).

Typically, the utility of noninvasive sensory feedback is validated by demonstrating that subjects can perform manual tasks with a prosthesis better with the feedback than without. In most cases, these demonstrations have involved amputees operating myoelectric bionic hands, with sensory feedback delivered to the residual limb. The ability to grasp and manipulate objects and to perceive their properties (size, shape, compliance) is improved with sensory feedback (407, 408, 414–418). The same approaches could in principle be applied in the context of BMIs for people with tetraplegia.

12.2. Sensory Substitution

Noninvasive sensory feedback typically involves the elicitation of touch sensations that are experienced where the stimulus is delivered rather than on the hand. As such, noninvasive tactile feedback—or at least any form that is available to individuals with tetraplegia—is a form of sensory substitution. Although such feedback improves prosthetics use on simple tasks in a laboratory setting, it is unclear whether this approach will generalize to activities of daily living (419). First, signals conveyed through sensory substitution require that attention be deployed to them, in contrast to natural sensory signals, whose integration into motor plans does not require attention. Second, sensory substitution can only convey information at a low bandwidth. Indeed, the devices themselves occupy large swaths of skin, and so only a limited number of them can fit over available skin territory. More importantly, the cognitive demands of the devices limits users’ ability to simultaneously take in information across more than a few channels at a time. Given the complexity of the hand and the limited bandwidth of sensory substitution, their potential to support dexterous limb behavior is dubious. In a rare study to test sensory substitution in the context of activities of daily living (ADLs) performed at home, noninvasive tactile feedback did not improve and in fact worsened performance on ADLs (420).

These inherent limitations notwithstanding, sensory substitution approaches can be improved by invoking basic principles of biomimicry. For example, the somatosensory representations of contact events in the intact nervous system emphasize contact transients relative to maintained contact (FIGURE 13). Sensory substitution devices that incorporate this feature of natural tactile feedback better support prosthetics-mediated interactions with objects than devices that do not (415).

12.3. Prognosis

As discussed above, for sensory feedback to improve performance requires a modicum of precision and accuracy from the control system. Poor control does not benefit from feedback, even under ideal conditions. Whether noninvasive feedback will provide significant benefits to noninvasive BMIs, which provide only rudimentary control, remains to be demonstrated.

13. A BROADER PERSPECTIVE ON BMIs

This article focuses on BMIs that restore reaching and grasping through closed-loop control of an artificial end effector or an individual’s paralyzed limb, where high-fidelity readout and write-in are required to extract moment-by-moment estimates of movement intention and provide sensory feedback that can improve performance on closed-loop tasks. At present, only invasive interfaces, in particular intracortical recordings, and perhaps electrocorticography, appear to provide the bandwidth necessary for such restoration. For example, in communication tasks, intracortical BMIs achieve orders of magnitude higher performance than noninvasive interfaces (38, 242). This difference is likely exacerbated for high-DOF control applications such as robotic arms; however, to our knowledge, there are no noninvasive BMI reports that provide standardized metrics (such as the functional clinical measures described above) that would allow direct comparison. Given the relative accessibility of noninvasive technologies, the lack of reports may itself be telling about the suitability of noninvasive interfaces for high-DOF applications.

We note, however, that alternate and emerging BMI applications are well suited for noninvasive interfaces. A key application is motor rehabilitation for individuals with stroke, where BMIs may complement conventional rehabilitation approaches (11). Rehabilitation BMIs read out movement intention and provide concomitant downstream neuronal activation—via visual neurofeedback, electrical stimulation, or exoskeleton-based movement of the hand—to elicit sensory signals with the goal of activating Hebbian-like plasticity mechanisms (339). Particular neural correlates of movement intention that are useful for rehabilitation, e.g., desynchronization of mu sensorimotor rhythms, can be detected robustly by using noninvasive interfaces such as MEG and EEG, even in individuals with paralysis who lack residual movements (11). Since an initial demonstration of the potential of an MEG-BMI to support stroke rehabilitation (421), several groups have demonstrated that BMI-based interventions can lead to improved recovery over standard rehabilitation (11, 384, 422–424). As stroke is one of the major worldwide causes of long-term motor disability, an effective BMI for neurorehabilitation could improve quality of life for millions of people.

13.1. Ethics of Implanted Devices

Although BMIs have potentially transformative impacts on human health, both research and current and future applications of BMIs raise a variety of ethical concerns, which merit careful consideration by researchers, companies, funding agencies, regulators, and end users.

Developing tools and therapies to alleviate the burden of neurological disorders and injury is widely considered to be an ethical imperative, which strongly motivates research and development (425–428). However, any implantable device involves some risk and, particularly for translational research, understanding and evaluating the sources of risk is critical for risk-benefit analysis, informed consent, evaluation by Institutional Review Boards, and protection of research participants from harm (427). Neurosurgery for device implantation or replacement raises the possibility of complications such as hemorrhage, stroke, or infection. People with partial motor impairments may be particularly susceptible to hard-to-quantify risks of damaging targeted brain areas that support their residual motor ability. Implanted devices also have multiple potential failure modes, including malfunction, infection, erosion, and lead migration, which can necessitate further corrective surgeries. Side effects or long-term consequences of electrical stimulation are hard to predict; for example, deep brain stimulation for movement disorders often results in unintended effects on speech and may have poorly understood side effects on mood, personality, cognition, or even perceptions of identity and agency. There are also potential financial risks, such as unanticipated costs for device maintenance and explantation. Furthermore, as implanted devices become capable of gathering ever-increasing quantities of personal or even intimate data, data privacy and security become important considerations.

Another tension in BMI development is the gap between researchers’ conceptions and the actual needs, priorities, and values of the end users (429). The large financial and time requirements involved in BMI research, combined with the small study populations, often limit researchers’ interactions with potential end users. From the end users’ perspective, lack of familiarity with emerging BMI technologies leads to a lack of awareness of how BMIs might fit into their everyday lives. The gap in knowledge and priorities underscores the importance of continued surveys of the end user population (13–15) and of direct interaction between researchers and people with motor impairments. As BMI technology matures, approaches such as user-centered design (429), which emphasizes understanding user experiences and involving users in the design and development process, may speed technology transfer and ensure that researchers create technologies that are actually used.

More broadly, the field faces ethical considerations that are unique to devices that harness and alter brain function. BMIs can potentially affect fundamental aspects of the human experience—identity, normality, authority, responsibility, privacy, and justice (430). Devices that restore function may restore a sense of identity and empowerment to people with impairment, but this restored function may also have unintended consequences. For example, if a brain-controlled device results in injury or damage to others, does the user bear sole responsibility? Furthermore, the past decade has seen unprecedented, world-changing events related to breaches of privacy and misuse of personal data, sometimes with little accountability. What are the consequences of privacy breaches when the data in question relate to peoples’ thoughts? Additionally, although BMIs may become wonderfully enabling, they are an extremely resource-intensive technology. How can we ensure that a lack of access does not exacerbate societal inequity? Moreover, society has unquestionably been affected by instant access to information (and misinformation) through the internet and mobile phones. What happens when this information can be directly input into the brain? Such issues can be expected to become even more prominent if the field shifts from a focus on rehabilitation to enhancement, as considered below. These and other issues make it imperative that bioethicists have a significant voice in shaping the field, and that regulatory frameworks are put into place to anticipate and control detrimental effects of BMI technology on society and human health (430, 431).

13.2. Beyond Sensorimotor Restoration

To date, BMI development has focused on restoration of lost function. In addition to the efforts to restore upper limb function discussed here, other efforts are underway to restore vision (432, 433), locomotion (434), and even memory (435, 445). However, the possibility that we might be able to flexibly interact with extracorporeal machines through neural interfaces has spurred efforts to use BMIs to enhance the abilities of able-bodied individuals. Early attempts involved piloting a fighter jet using brain signals or sensing infrared light through a brain interface (318) with some measure of success. These were relatively simple tasks, and performance was much poorer than that mediated by limbs or infrared goggles. Nonetheless, the premise underlying BMI-mediated enhancement is that not only might we be able to interact more flexibly through a BMI than we can through our peripherals (limbs, senses), artificial signals might be multiplexed along with the natural ones—on both the input and output sides. The same pools of neurons might then be engaged in different functions, perhaps even at the same time. For now, the bandwidth of current BMIs is far too limited to allow for control of a complex extracorporeal effector through brain signals or to convey feedback in a way that would complement our existing sensory systems. Restorative BMIs are predicated upon the principle of biomimicry to control complex anthropomorphic arms or interpret multidimensional sensory signals. This technology leverages the fact that we learn during childhood to control our bodies and interpret sensory signals, thereby forming and honing the sensorimotor representations of our bodies. As there exist no neuronal representations of arbitrary extracorporeal effectors, native neuronal representations cannot be leveraged. Whether brains—especially adult brains—can learn to flexibly control complex and arbitrary extracorporeal devices remains to be demonstrated (436). The principles that underlie BMIs might also be invoked to have brains communicating directly with one another (437). One brain can send electrical signals to another brain, which would then interpret those signals. In principle, the two brains could thus communicate with much greater flexibility over a greater bandwidth. As is the case for other BMIs, however, this connection between brains is currently so primitive that it is dwarfed by the more traditional means of brain-to-brain communication, namely language.

An interesting open question is whether humans even have the capacity to communicate with substantially higher information rates, or whether biology imposes fundamental limits on our capacity to process and transmit information. For instance, recent work surveyed information transfer rates for speakers of 17 different languages and found that although the amount of information conveyed per syllable showed substantial variation across languages (by more than a factor of 2), these differences were counterbalanced by corresponding differences in speech rates across languages (438). Thus, information transfer rates showed remarkable consistency across languages (∼39 bits/s), despite the potential for faster rates in languages with higher per-syllable content. One interpretation is that this consistency may reflect underlying limits in the ability of the human nervous system to process information. Alternatively, we may one day find that such limits in part reflect constraints imposed by the communication channels.

Although the current technology is far too primitive to support enhancement BMIs, the private sector is beginning to invest in BMI technology, which is liable to lead to great strides therein. Imagine if we could one day inhabit each other’s minds or access the whole of human knowledge stored in the cloud as we can our own knowledge!

GRANTS

C.P. is supported by National Science Foundation Grant NCS-FO 1835364, Defense Advanced Research Projects Agency Contract PA-18-02-04-INI-FP-021, an Alfred P. Sloan Foundation research fellowship in Neuroscience, a Eunice Kennedy Shriver NICHD K12 fellowship (K12HD073945), and the Simons Foundation. S.J.B. is supported by NINDS Grants NS107714, NS095251, and NS122333.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

C.P. and S.J.B. drafted manuscript; edited and revised manuscript; and approved final version of manuscript.