Restoration of sensory information via bionic hands

Abstract

Individuals who have lost the use of their hands because of amputation or spinal cord injury can use prosthetic hands to restore their independence. A dexterous prosthesis requires the acquisition of control signals that drive the movements of the robotic hand, and the transmission of sensory signals to convey information to the user about the consequences of these movements. In this Review, we describe non-invasive and invasive technologies for conveying artificial sensory feedback through bionic hands, and evaluate the technologies’ long-term prospects.

The hand allows for a wide array of interactions with objects. For example, grasping involves a range of behaviours, from precision grip with the index finger and thumb, to power grasp with all of the fingers and the palm. The hand’s versatility is partly enabled by its anatomical complexity (it comprises many joints and is driven by many muscles¹). Hand control also requires a sophisticated neural system to configure the digits in task-appropriate ways and to apply finely graded forces on objects. Also important are sensory signals from the hand that convey information about the hand’s state (movements and posture) and about its interactions with objects (contact timing, location and pressure).

Losing hand function can cause severe physical incapacity and even mental disabilities. Each year, thousands of people suffer the consequences of upper-limb paralysis² or of amputation³ caused by a traumatic event or disease. Bionic (robotic) limbs can restore independence to these individuals. The development of such prostheses involves methods that infer intended movements from signals acquired from the user (neural or otherwise), and the execution of these motor intentions by the prosthetic device. There are a variety of technologies for the extraction of control signals from the user’s muscles, nerves or brain, or from the user’s residual movements⁴.

However, the performance of unidirectional efferent neuroprosthetic systems is limited by the inadequate sensory information available to the user: movements are guided mainly using vision⁵, even if proprioceptive signals stemming from the residual forearm muscles used for myoelectric control of the bionic hand, and sound from the robotic actuators (and other incidental cues), may also contribute⁶. Instead, in non-disabled individuals, limb control relies heavily on somatosensory signals that track the state of the limb and its interactions with objects. Nerve fibres that innervate the muscles, tendons, joints and skin convey information about the posture and movements of the limb and about the forces they exert. Nerve fibres that innervate the skin convey information about the initiation and termination of contact with an object, about which parts of the hand make contact with the object, about the forces exerted at each location, and about the object itself (its size, shape and texture^7,8). Tactile signals allow us to maintain contact with an object without having to attend to it visually. Touch and proprioception are also critical to our ability to dexterously manipulate objects, as evidenced by the deficits that result when we lose these sensory signals. For example, we rely on our sense of touch to apply just enough force on objects so as to not drop them; when fingertips are numbed with a local anaesthetic, we exert much more force than is necessary⁹. When de-afferentation is complete and includes both touch and proprioception, as is the case in some peripheral neuropathies, dexterity is completely abolished and hand use is clumsy and effortful, even with an otherwise intact motor system¹⁰.

The realization that prosthetic control is limited by the overreliance on visual feedback, and the recognition that somatosensation plays an important psychosocial role, have led to efforts aimed at restoring tactile and proprioceptive feedback in bionic limbs. Some strategies involve mechanically or electrically stimulating the skin to convey information about the state of the limb (such as posture and movements) and about its interactions with objects (see Fig. 1a and the ‘Non-invasive sensory feedback’ section). Other strategies involve surgically implanting devices that interface with peripheral nerves or with the brain to directly activate neurons by injecting small currents into them (see Fig. 1b,c and the ‘Interfaces of the peripheral nervous system’ and ‘Interfaces with the central nervous system’ sections). The different technologies entail different costs and benefits, and are suitable to different degrees for different user populations, such as amputees or individuals with tetraplegia.

Fig. 1 |. Technologies for restoring sensory feedback via bionic hands.

a, Non-invasive sensory feedback. The output of sensors on the bionic hand drives mechanical or electrical stimulation of the skin to convey information about contacted objects, in this case through stimulators housed in an armband. b, Sensory feedback via an electrical interface with the peripheral nervous system. Sensors on the bionic hand drive the electrical stimulation of a nerve to elicit sensations referred to the phantom hand. c, Sensory feedback via an interface with the central nervous system. Sensors on the bionic hand drive the electrical stimulation of the central nervous system to elicit sensations referred to the de-afferented hand. Credit: Image courtesy of Kenzie Green

Ideally, sensory restoration should reproduce normal patterns of neuronal activation. This would make the resulting sensations completely natural and intuitive. However, state-of-the-art technology does not confer the ability to reproduce the complex patterns of neuronal activation evoked along the intact somatosensory neuraxis during activities of daily living. Nonetheless, biomimicry is a main driving principle of artificial sensory feedback: to evoke patterns that are, in some relevant way, similar to the natural patterns of neuronal activation^11–18. Although individual neurons may not be activated in a natural way, key aspects of the aggregate response of populations of neurons may be sufficiently similar, in a manner that conveys useful information about limb state or about objects in contact with it. A simple example is that of somatotopy: when our thumb touches an object, we experience a sensation on the thumb. Some approaches to sensory feedback preserve this mapping, so that contact with the prosthetic thumb induces a sensation experienced on the phantom thumb (in amputees) or on the de-afferented thumb (in individuals with tetraplegia). Such biomimetic mapping is intuitive and therefore does not require the user to learn it. Hence, biomimicry entails conveying sensory information by using the nervous system’s ‘native language’. Nerves encode different object features in different ways. For example, local contours and coarse textural features are encoded in spatial patterns of afferent activation¹⁹, whereas fine textural features are encoded in temporal patterns of activation elicited as a finger is run across a surface²⁰. To the extent that the nerve is activated in a natural way, these various neural codes will be engaged, supporting haptic perception and object manipulation. Instead, sensory substitution—an alternative strategy—conveys information via a completely novel sensory stream. Biomimetic feedback may be preferable (as in principle it requires less learning on the user’s part), yet it may not always be possible or it may have costs (such as surgery) that overshadow its advantages. In such cases, sensory substitution may be preferred.

Non-invasive sensory feedback

Non-invasive strategies to convey sensory feedback interface with the user’s body without the need for surgical intervention. Although non-invasive interfaces have limited potential to provide biomimetic sensory feedback, the resulting sensory substitution may be sufficient to support some degree of dexterity. Non-invasive technologies for conveying sensory feedback through bionic hands, first introduced in the 1970s^6,21,22, can be categorized by the nature of the stimulation strategy: electrical or mechanical.

Electrotactile transcutaneous electrical nerve stimulation.

Electrotactile stimulation involves injecting electrical currents through the skin, thereby activating tactile nerve fibres. The sensations evoked by the technique, which was introduced to convey sensory feedback in bionic hands in the 1980s²³, are unnatural—they are typically described as ‘electricity’, ‘vibration’ or ‘tingling’^24,25—and are often unpleasant²⁶. Electrotactile stimulation does not typically produce sensations that are referred to the hand, so the sensations are not somatotopic; however, users can learn to interpret non-somatotopic electrotactile feedback with a few days of training²⁷. In some cases, the stimulation of the stump evokes sensations that are referred to the (missing) digits, such that different patches of stump skin evoke sensations on different digits²⁸ (Fig. 2a). This organization can in principle be exploited to convey somatotopic feedback. Mixed stimulation strategies involving the modulation of the spatiotemporal pattern of stimulation can improve task performance over simpler strategies that involve the modulation of spatial or temporal patterns individually²⁹. Although electrotactile feedback has been shown to support haptic object recognition (that is, stereognosis; a complex sensory task)³⁰, its utility has not been demonstrated outside of the laboratory.

Fig. 2 |. Somatotopic mapping following targeted reinnervation.

a, Stimulation of different patches of skin on the stump evokes sensations referred to different digits. The mapping can in principle be exploited to achieve tactile feedback that is more intuitive. b, Mechanical stimulation of the chest, to which nerve fibres have been rerouted. The stimulation elicits sensations referred to the phantom limb, and often to the phantom hand¹⁷⁴. Credit: Image courtesy of Kenzie Green; panels adapted with permission from ref. ²⁸, Foundation of Rehabilitation Information (a) and ref. ⁴², Elsevier (b)

As with electrotactile stimulation, transcutaneous electrical nerve stimulation (TENS) involves the delivery of electrical stimulation to the skin. However, rather than targeting nerve fibres that locally innervate the skin, the device is placed over nerves that innervate more distal skin locations, thereby evoking sensations that are referred to these locations^25,31. In amputees, TENS can be used to evoke localized sensations that are referred to the amputated limb^32,33. The resulting sensory feedback improves the user’s ability to perform tasks such as object discrimination and grip force modulation with a bionic hand, sometimes to levels comparable to those achieved with direct stimulation of the nerves^34,35. However, as with electrotactile stimulation, sensations evoked by TENS are unnatural—a limitation that may be mitigated with stimulation strategies that are more biomimetic³⁶.

Feedback based on electrical stimulation offers two main advantages: the electrodes are lightweight and can be easily integrated into the prosthesis socket, and electrical stimulation consumes little power compared with its mechanical counterpart. However, the efficacy of electrotactile stimulation is strongly dependent on skin moisture, which fluctuates wildly. This instability can be mitigated by monitoring skin impedance and implementing stimulation regimes that account for it³⁷. Another issue is that electrical fields interfere with the myoelectric measurements required for prosthesis control³². This electrical interference is circumvented by interleaving stimulation and recording³⁸.

Mechanotactile stimulation.

Mechanotactile stimulation results in natural sensations that can in principle convey intuitive sensory feedback. For example, pressure applied to the prosthetic finger can be conveyed by applying pressure on the skin^39,40. Mechanotactile feedback has been used to convey information about grasp force⁴¹, the onset and offset of contact³⁹, and the location of contact⁴⁰. The main drawback of mechanotactile feedback is that mechanotactile stimulators are difficult to miniaturize and are typically power-hungry. These limitations have hindered the deployment of such devices in a clinical context.

Mechanotactile feedback has also been used in patients undergoing targeted reinnervation⁴²—a surgical approach that reroutes the residual nerves to the skin of the residual limb or to the upper torso. Some of the nerve fibres then reinnervate the cutaneous mechanoreceptors so that tactile stimulation activates the rerouted nerve fibres and elicits sensations that are often referred to the missing limb (Fig. 2b). This approach has been deployed in a closed-loop hand prosthesis⁴², but interactions between the muscle signals used to control the device and the mechanotactile stimulation delivering feedback over the same patch of skin may limit the long-term prospects of the technology. In principle, electrotactile stimulation can also be used with targeted muscle reinnervation.

Vibratory feedback.

Mechanotactile feedback can also be delivered using commercially available vibrators (Fig. 3a), such that vibratory amplitude or frequency is modulated according to hand state, contact parameters or object properties. Vibratory feedback has been used to convey information about hand aperture⁴³, grasp force^44,45, grasp speed⁴⁶, object compliance⁴⁵ and surface texture⁴⁷. However, in all cases the users need to learn to interpret this feedback, as the mapping between the contact dynamics or object properties and the resulting sensory experience is arbitrary. Yet vibratory feedback can in principle convey rich sensory information by modulating the different stimulation parameters. For example, hand-opening and hand-closing speeds have been conveyed by modulating the stimulation amplitude at different frequencies⁴⁶.

Fig. 3 |. Examples of non-invasive stimulation.

a, Array of coin motors used to deliver vibrotactile feedback. b, Configuration of a combination of electrotactile and vibrotactile stimulators. Credit: Image courtesy of Kenzie Green; panels adapted with permission from ref. ²⁶, IEEE (a) and ref. ⁵⁶, IEEE (b)

Vibratory feedback has also been used to signal transitions between sub-goals of tasks, on the basis of the framework ‘discrete event-driven sensory feedback control’⁴⁸ (DESC). When users were provided information about the onset and offset of object contact through vibratory bursts, their performance improved on various object-manipulation tasks that simulate activities of daily living⁴⁹. For example, in the ‘virtual egg’ test, the user is tasked with moving a fragile object from one location to another; when too much force is exerted on it, the object breaks. This task requires the user to exert force within a narrow range: enough to pick the object up, but less than the force needed to break it. DESC-based vibratory feedback enabled users to move more eggs in the allotted time (and while breaking fewer of them) than when using no vibratory feedback, or when using vibrations that were maintained throughout the interaction⁴⁹.

As with mechanotactile feedback, vibratory feedback is limited by the bulkiness, rigidity and power consumption of the vibrators, which rely on electromagnetic eccentric motors, electrostatic piezoactuators or electroactive polymer-based actuators. One possible path forward is to develop soft pneumatic actuators^50,51, which are lighter and less power-hungry⁵².

Vibratory stimulation of the tendon can be used to evoke proprioceptive sensations in amputees^53,54. This may enable non-invasive artificial proprioception; however, the imposed vibrations contaminate the signals from the muscles that are monitored and used to control the prosthesis. This problem may be mitigated by implanting the vibrators on the tendon⁵⁵. Still, tendon vibration triggers muscular reflexes, which are also likely to contaminate the control signals.

Integration of vibratory and electrotactile feedback.

Vibrotactile and electrotactile stimulation can be combined to deliver sensory feedback⁵⁶, thereby providing two simultaneous and partially independent streams of information (Fig. 3b). Users can discriminate the intensities of the stimulus in each modality (for example, three vibrotactile and three electrotactile; nine total stimulation sites) with limited cross-modal interference.

Interfaces with the peripheral nervous system

Non-invasive technologies for the restoration of sensory feedback require the activation of somatosensory nerves through the skin. This limits the spatial resolution, owing to the spread of electrical current or to restrictions on the density of tactors. The limited spatial resolution in turn limits the bandwidth of achievable sensory signals. To overcome the limitations of non-invasive strategies, nerve fibres can be directly activated via an electrical interface.

Sensory innervation of the hand.

The glabrous skin of the hand—its palmar surface—is innervated by about 12,000 tactile nerve fibres⁵⁷. These nerve fibres fall into four classes, each driven by a different mechanoreceptor and responding to a different type of skin deformation (Fig. 4a). Slowly adapting type-1 (SA1) fibres, which innervate Merkel disks, have small receptive fields and are sensitive to static deformations or to slow skin deformations. SA1 fibres are sensitive to edges and local object contours⁵⁸. Slowly adapting type-2 (SA2) fibres, which innervate Ruffini cylinders, have large receptive fields and respond preferentially to skin stretch. SA2 fibres may play an important role in conveying information about the lateral loads imposed on the fingertip. Rapidly adapting (RA) fibres, which innervate Meissner corpuscles, have small receptive fields and are primarily sensitive to changes in skin deformation. RA fibres are sensitive to vibrations of 20–100 Hz in frequency and to textured surfaces scanned across the skin. Fibres innervating Pacinian corpuscles (PCs) have large receptive fields and are exquisitely sensitive to high-frequency vibrations. PC-associated fibres play an important role in fine texture perception and in sensing through tools. SA1 and RA fibres terminate preferentially in the fingertips, and their density decreases towards the arm and beyond. PC fibres show a much shallower gradient in the opposite direction, with greater density on the palm. SA2 fibres are most common around the nail and are rare in the glabrous skin.

Fig. 4 |. The somatosensory periphery.

a, The different types of low-threshold cutaneous mechanoreceptors that innervate the palmar surface of the hand. Clockwise from top left: Meissner corpuscle, Ruffini ending, Merkel disk and Pacinian corpuscle. b, The hand is innervated by the median nerve (blue), the ulnar nerve (light green) and the radial nerve (purple). c, Nerves are divided into individual bundles of fibres (known as fascicles). Credit: Image courtesy of Kenzie Green

The electrical stimulation of individual fibres of different types systematically evokes different sensations⁵⁹: activating a single SA1 fibre gives rise to a sensation of pressure; activating an RA fibre gives rise to a sensation of skin flutter (low-frequency vibration); and activating a PC fibre gives rise to a sensation of high-frequency skin vibrations. However, electrically activating individual SA2 fibres does not typically evoke a sensation.

Structure of the somatosensory nerves.

Nerve fibres that innervate the various mechanoreceptors in the glabrous skin of the hand converge to form three nerves (Fig. 4b): the median nerve, which innervates the first three digits and half of the fourth digit, along with the lateral aspect of the palm; the ulnar nerve, which innervates the fifth digit, the other half of the fourth digit, and the medial aspect of the palm; and the radial nerve, which innervates some of the lateral palm under the first digit (and mostly the dorsal surface of the hand). Within the nerves, fibres bunch into fascicles, which are largely electrically isolated from one another (Fig. 4c). Each fascicle typically carries nerve fibres that innervate a common patch of skin.

Interfaces with the somatosensory nerves.

Limb loss results in the loss of the mechanoreceptive end organs; however, the somatosensory nerves and downstream pathways conveying signals from those organs to the central nervous system remain functional^60,61. Activating the nerves thus engages these pathways and elicits tactile and proprioceptive sensations. Approaches requiring surgical intervention to stimulate the somatosensory nerves, first established in 1974, place single-channel extraneural cuff electrodes around the median nerves for the upper extremities⁶² and around the femoral nerve for the lower extremities⁶³. The evoked sensations are referred to the phantom limb and are generally diffuse, unstable and described as paresthesias. Interfaces with the peripheral nervous system can be categorized according to their level of invasiveness (how much do they penetrate the nerve?), which in turn impacts their selectivity (can they activate a small group of fibres at the exclusion of others?).

Extraneural interfaces surround a peripheral nerve but do not insert elements within the nerve, causing the least disturbance to neural tissue. As a result, these interfaces tend to be particularly robust^18,64–66. Indeed, circular electrodes that mechanically conform to the nerve (such as the spiral electrode⁶⁷ and the helical electrode⁶⁸) have been stable for decades in clinical applications^69–71. A drawback of the circular configuration is that it provides little surface area to interact with neural tissue. To increase the surface area and proximity to neural tissue (without penetrating the nerve), the flat interface nerve electrode (FINE)⁷²—another class of extraneural electrode—compresses the nerve, allowing for selective stimulation of individual fascicles and possibly sub-fascicles^72–74 (Fig. 5a). FINEs implanted in the median, ulnar and radial nerves¹⁸ have been shown to yield stable sensitivity to stimulation for years⁶⁴.

Fig. 5 |. Electrical interfaces with the peripheral nerves.

The panels show three examples of chronically implanted interfaces. a, The flat interface nerve electrode. b, The transverse intrafascicular multichannel electrode. c, The Utah slant electrode array. d, The schematic indicates how the three electrode arrays interact with the nerve. Credit: Image courtesy of Kenzie Green

In intraneural interfaces, electrical contacts penetrate the nerve. The most used configurations are the longitudinal intrafascicular electrode (LIFE)^75,76, which has a lead comprising multiple contacts that proceeds along the nerve; the transverse intrafascicular multichannel electrode (TIME)⁷⁷, whose multi-contact lead is implanted through the thickness of the nerve (Fig. 5b); and the Utah slanted electrode array (USEA)⁷⁸, which consists of 100 shanks arrayed over a 4 × 4-mm area, with lengths progressively longer along the nerve’s distal–proximal axis (Fig. 5c). The self-opening neural interface (SELINE) is a three-dimensional version of the TIME interface⁷⁹ that consists of a main shaft flanked by ‘wings’ designed to improve the implant’s stability. Intraneural interfaces can stimulate more selectively because the electrical contacts are embedded in the nerve and are contiguous with nerve fibres. They can remain stable for at least six months^80,81, but chronic stability over longer periods has not been tested.

Another way to achieve an interface with the nerve is to have it grow into the stimulating channels, thereby achieving an intimate connection between the electrode and neural tissue^82–85. However, the development of this technology is at an early stage. Alternatively, stimulation can be applied to the lateral spinal cord where the sensory nerve enters the cord. This has been shown to elicit tactile percepts referred to spatially restricted regions of the phantom hand (such as a finger⁸⁶). The advantage of this approach is that it can be applied to individuals whose residual nerve has been (nearly) completely destroyed, as is the case after shoulder disarticulation.

The sensory consequences of electrical stimulation of the somatosensory nerves.

The sense of touch conveys rich information about objects that we grasp and manipulate. Object interactions activate hundreds or thousands of tactile nerve fibres, each in a different way depending on its type (SA1, SA2, RA or PC) and on its location on the hand relative to the points of contact with the object^87–89. Full restoration of touch would require the activation of each of these fibres independently with an idiosyncratic pulse train—a feat that cannot be achieved in the foreseeable future¹⁴. Hence, sensory restoration is limited by the small number of stimulating channels relative to the number of nerve fibres. Rather than attempting to reproduce naturalistic patterns of activation in complete populations of nerve fibres, the goal is to produce patterns of neuronal activation that are useful for object manipulation and that promote the embodiment of the bionic hand. The common way to characterize the artificial sensations evoked by the electrical stimulation of the peripheral nerve is to assess their location (where is the sensation experienced?), their intensity (how strong is the sensation?), and their quality (what does it feel like?). Also, a ubiquitous phenomenon in perception is that of adaptation: our sensory systems adjust their sensitivity according to ambient levels of stimulation (for example, we are more sensitive to light at night than during the day, and without such adjustments we would not be able to see in twilight or in the midday sun).

The electrical stimulation of an individual nerve fibre produces a sensation that is referred to a specific location on the skin, which itself corresponds to the location of the fibre’s receptive field⁵⁹ (Fig. 6). For example, activating an RA fibre with a receptive field on the index fingertip will induce a sensation (of flutter) that is experienced on the index fingertip. After amputation, activating a nerve fibre in the residual nerve that originally innervated a specific location on the hand produces a sensation that is referred to that location. This phenomenon can be exploited by adopting somatotopic mapping to convey information about where the bionic hand makes contact with an object. For example, the sensor on the index fingertip of the bionic hand can drive stimulation of a nerve fibre or nerve fascicle with a receptive field on the index fingertip, evoking a sensation referred to the index fingertip and thereby intuitively signalling the contact location to the user. As neighbouring fascicles have neighbouring receptive fields^90–97, increasing the stimulation strength (and thereby recruiting nearby fascicles) typically results in an increase in the area over which the sensation is experienced, rather than in the emergence of sensations randomly located on the hand^18,98. Information about contact location is thus robust to changes in stimulation intensity. And because functionally important regions of the hand (such as the fingertips) are more densely innervated, electrical stimulation tends to evoke sensations referred to these regions. Interestingly, the sensation seems to ‘move’ from the stump to the hand over time, suggesting a ‘reawakening’ of dormant sensory representations after re-afferentation^81,99. However, once this reawakening has taken place, the location of the projected field of each stimulating lead is stable^81,100–102.

Fig. 6 |. Projected fields for nerve stimulation.

The electrical stimulation of the residual somatosensory nerves of an amputee evokes localized and stable percepts on the missing hand. Stimulation through different electrodes located on the median (blue), ulnar (green), and radial (red) nerves over the course of two months led to consistent perceived sensations on the phantom hand (the locations are indicated by the coloured patches). Credit: Image courtesy of Kenzie Green; adapted with permission from ref. ¹⁸, AAAS

All tactile sensations fall along a continuum of intensity: some sensations are faint, while others are strong. For example, the sensory experiences of pressure¹⁰³ or of vibratory amplitude¹⁰⁴ fall along a common intensive continuum. The sense of touch endows us with an exquisite sensitivity to contact pressure, which allows us to exert just enough force on an object to pick it up⁹. As the depth of skin indentation increases, populations of tactile nerve fibres become increasingly active, and the perceived intensity is determined by the strength of the response (the firing rate) evoked in populations of tactile nerve fibres¹⁰⁴. The intensity of an artificial tactile sensation can thus be modulated by manipulating the population firing rate in the nerve. Indeed, increasing stimulation frequency (which leads to an increase in the firing rate of activated fibres) or stimulation amplitude (which leads to a recruitment of additional fibres) results in an increase in the population firing rate and thus in an increase in perceived intensity^18,65,105. In fact, the population firing rate is approximately linearly related to the activation charge rate (that is, the amount by which the stimulation current exceeds a threshold), which is in turn linearly related to perceived intensity¹⁷, as gauged through subjective reports of intensity. Sensitivity to changes in intensity (either mechanical or electrical) can be probed by having the users discriminate between stimuli that vary in intensity. The ‘just-noticeable difference’ (that is, the amount by which the stimulation intensity needs to change for the user to reliably detect the change) is an index of sensitivity that can then be used to estimate how many increments in intensity can be signalled across the safe range of intensities. Current neural interfaces with the peripheral nervous system allow for dozens of just-noticeable differences, which leads to improved use of prosthetics^18,34,81,106.

Tactile sensations do not just differ in their perceived location or intensity. One tactile experience may feel like a velvety surface moving across the skin; another may feel like the edge of an object indenting the skin. The quality of a sensory experience is fine-grained, and the resulting perceptual space is rich. Even texture (a subspace of touch) defies full characterization¹⁰⁷. Although location and intensity each rely on a single neural code (the activated fibre or fascicle for location, and the population firing rate for intensity), the neural underpinnings of perceptual quality are much more elusive. In fact, the exact quality of a sensation is determined by the precise spatiotemporal pattern of activation across the entire neuronal population. For example, the sensation of a velvety surface moving across the skin involves the activation of many nerve fibres, each with its idiosyncratic response. And the indented edge evokes a completely different pattern of activation in an overlapping afferent population. Sensory quality is important because, beyond location and intensity, it carries critical information about the stimulus (is it velvet, or a sharp edge oriented perpendicular to the long axis of the finger?). Because of technical limitations in the reproduction of neuronal patterns with such a level of precision and selectivity, the current challenge is to convey critical stimulus information. One way to alter the quality of an electrically evoked sensation is to change the frequency of stimulation, up to about 50 Hz—changes in frequency beyond this range only impact the perceived intensity¹⁰⁸.

Electrical stimulation of the nerve with a constant pulse train results in sensations that are typically described as a ‘tingle’ or ‘electrical’¹⁸. One main limitation in the systematic manipulation of the quality of artificial tactile sensations is that electrical stimulation results in the synchronous activation of a number of nerve fibres near the electrode tips, regardless of fibre type. This pattern of activation is, of course, unnatural (as different nerve fibres respond differently to natural skin deformations). Yet electrical interfaces can produce aggregate patterns of nerve activation that are naturalistic: in other words, whereas individual nerve fibres may not be activated in a natural way, populations of nerve fibres can be activated in ways that preserve some essential features of the natural aggregate activity¹¹. For instance, whereas constant pulse trains (constant amplitude and frequency) evoke parasthetic sensations, pulse trains with varying amplitude or frequency (or both) evoke sensations that are less unnatural¹⁸, perhaps because the aggregate pattern of nerve activity is more natural under these stimulation conditions. When the stimulation is biomimetic—that is, explicitly designed to produce patterns of nerve activation that are more naturalistic—the resulting sensory percepts are perceived as more natural, but this effect is limited¹⁰⁶. The main impact of biomimetic stimulation is to improve the dexterity conferred to the bionic hand.

Extended exposure to an intense stimulus reduces sensitivity to it. Such adaptation—which tunes the sense of touch to ambient levels of stimulation and increases its sensitivity to changes in stimulation^109,110—is also observed in artificial touch. Indeed, prolonged electrical stimulation of the nerve leads to decreased sensitivity, and the degree and time course of this desensitization matches that of natural touch¹¹¹. Adaptation can become problematic for low-amplitude stimulation regimes that activate only a handful of fibres because their responses can become ‘adapted out’, leading to a complete abolition of the sensation within seconds¹¹².

The impact of sensory feedback on haptics and on object manipulation.

The characterization of the sensory consequences of electrical stimulation provides insights that can help guide the design of sensory-feedback algorithms. A main goal of artificial somatosensation is to improve the functionality of bionic hands. One way to gauge the functional benefits of sensory feedback is to assess how well users can distinguish objects explored with the prosthesis. Sensitization of the bionic hand can enable users to sense the size and compliance of objects^34,113. This task is particularly challenging when the users are controlling the prosthetic hand, because the sensory signals evoked by each object vary from trial to trial, owing to variations in the exploratory movements of the hand. Hence, the sensory signals must be interpreted in the context of the movements of the prosthesis. Texture can also be sensed through a prosthesis, as shown with the discrimination of spatially varying gratings¹⁶.

Another way to test the functional benefits of sensory feedback is to assess whether it improves object manipulation. One approach consists of instructing blindfolded users to perform a so-called ‘box and blocks’ task: picking up a block in one box and placing it in another. Performance is judged by how many blocks the user can move within a fixed period of time. Without vision or tactile feedback, users perform very poorly on this task. When tactile feedback is provided, performance improves substantially¹¹⁴. However, the box and blocks task does not constitute a stringent test of sensory feedback because, when vision is present, artificial touch improves performance only modestly. A more sensitive test of sensory feedback involves the manipulation of fragile objects via the virtual egg test. Performance on this task is significantly improved with sensory feedback, even when the user can see the bionic hand^81,113, because contact force cannot be gauged visually. With biomimetic sensory feedback, performance on this task improves even further^106,113. Sensory feedback also enabled the plucking of stems from cherries using a bionic hand¹⁸—a task that is almost impossible to accomplish without touch (because of the fruit’s fragility).

Embodiment.

Artificial sensory feedback also enhances the embodiment of the prosthetic hand: the simultaneous visual experience of the bionic hand touching an object and the tactile experience of the object induces the sensation that the prosthesis is part of the body¹¹⁵, a phenomenon that is closely related to the ‘rubber hand illusion’¹¹⁶. Embodiment of the bionic hand improves the user’s acceptance of the prosthesis, as it confers the feeling of wholeness that was lost with the amputation^101,117. Embodiment is bolstered when the sensations are made more natural through biomimetic stimulation¹⁰⁶, or when visual feedback explicitly signals contact¹¹⁸.

Phantom pain.

Amputation often gives rise to phantom pain—that is, chronic pain experienced as originating from the missing limb¹¹⁹. In some cases, phantom pain is so severe as to be debilitating. Providing sensory feedback through an interface with the peripheral nervous system seems to alleviate the symptoms of phantom pain^{18,65,118,120}, but this remains to be replicated in systematic and long-term placebo-controlled studies. The long-term use of a sensorized bionic hand may further reduce the symptoms of phantom pain because the restoration of tactile signals may reverse changes in the central nervous system brought about by amputation.

Restoration of proprioception.

The sense of the movement and position of our body in space—known as proprioception—is mediated by nerve fibres that innervate the muscle spindles¹²¹, and possibly by nerve fibres that innervate the skin around joints¹²². Electrical stimulation of the nerve rarely evokes a proprioceptive sensation without also activating muscles^123,124, except when stimulating a small number of fibres¹²⁵. Sensory substitution can convey proprioceptive information about limb conformation (typically hand aperture) by stimulating a channel that evokes a cutaneous sensation^18,124. The user can then learn to use this sensory channel to infer the state of the limb, and use the information in combination with tactile feedback to perform simple tasks¹²⁶. Whether this approach can support activities of daily living remains to be shown.

Interfaces with the central nervous system

Electrical interfaces with peripheral nerves cannot be applied to patients with upper spinal cord injuries, for whom the connection between the nerves and the central nervous system has been severed. Although there have been attempts to implant arrays of electrodes in the cuneate nucleus^127–129 and in the ventral posterolateral thalamus^130–133, the most accessible and explored electrical interface with the central nervous system involves the somatosensory cortex, which lies at the surface of the brain or near it (Fig. 7a).

Fig. 7 |. Electrical interfaces with the central nervous system.

a, Possible loci of neural interfaces for the restoration of touch in the central nervous system. From bottom to top: the cuneate nucleus, the thalamus and the somatosensory cortex. The insets indicate the arm and face regions of the somatosensory homunculus in the human somatosensory cortex (top left), and the cortical fields in the Brodmann’s areas 3a, 3b, 1 and 2 of the somatosensory cortex (top right). b, Surgical image showing the implantation, in a human, of two arrays in the motor cortex (anterior to the central sulcus; left) and of two arrays in the somatosensory cortex (posterior to the central sulcus; right)¹⁴⁶. The coloured areas indicate the mapping of the palm, thumb, index finger and little finger in the somatosensory cortex. The symbol ‘Ω’ indicates the presumptive location of the hand in the motor cortex (the so-called ‘hand knob’). Credit: Image courtesy of Kenzie Green; adapted with permission from ref. ¹⁴⁶, AAAS

The somatosensory cortex.

Located at the anterior edge of the parietal lobe, the somatosensory cortex comprises four cortical fields: the Brodmann’s areas 3a, 3b, 1 and 2 (ref. ¹³⁴). Neurons in area 3a respond primarily to joint movements, neurons in areas 3b and 1 respond to light touch, and neurons in area 2 exhibit both response properties. Somatosensory fields are organized hierarchically: areas 3a and 3b constitute the earliest stages of proprioceptive and tactile processing, respectively; area 1 is the next stage of cutaneous processing; and area 2 receives input from areas 3a and 1. Receptive fields tend to become larger and neurons exhibit increasingly complex response properties along the somatosensory neuraxis. For example, neurons that are strongly selective for motion direction are much more prevalent in area 1 than they are in area 3b¹³⁵. Each cortical field comprises a map of the body, laid out from feet to head, proceeding anterolaterally along the central sulcus¹³⁶. Within this body map and along this axis, the hand representation is also systematically laid out, from the little finger to the thumb (Fig. 7a). In macaques, the hand representation spans approximately 6–9 mm along the central sulcus and 10 mm along the orthogonal axis¹³⁶. In humans, this representation is nearly tenfold larger¹³⁷.

Early studies.

It has long been known that electrical stimulation of the somatosensory cortex evokes tactile and proprioceptive sensations. When searching for the focus of seizures in patients with epilepsy, electrical stimulation delivered to the surface of the somatosensory cortex was found to evoke tactile percepts whose location on the body varied systematically with the cortical location at which electrical stimulation was applied: stimulation near the mid-line tended to evoke sensations in the lower body, and stimulation near the lateral sulcus evoked sensations on the arms and face. This led to the discovery of the somatosensory homunculus¹³⁸ (Fig. 7a). Later, experiments in monkeys showed that changing the frequency of intracortical microstimulation (ICMS) resulted in systematically discernible percepts^139,140. These early studies established that the location and quality of an ICMS-evoked sensation could be systematically manipulated, and laid a foundation for the restoration of touch via ICMS of the somatosensory cortex.

Electrical interfaces with the somatosensory cortex.

The somatosensory cortex can be electrically interfaced through chronically implanted electrode arrays which pierce through the membranes that enfold the brain (the dura mater, the arachnoid and the pia mater) such that the electrode tips impinge upon the grey matter, where the cell bodies of cortical neurons are located. Two types of implants have been most commonly used in experiments in humans and in non-human primates: Utah electrode arrays (UEAs; commercialized by Blackrock Microsystems), which consist of a 10 × 10 array of electrodes, each 1–1.5-mm long, covering 4 × 4 mm of cortex (Fig. 7b); and floating microelectrode arrays (FMAs; commercialized by Microprobes for Life Sciences), which consist of 32 electrodes arrayed over a 4 × 1.8-mm area of cortex. In both types of array, the electrodes are coated with an insulating material up to (but not including) the tip, so that small currents can be injected at the tip. In UEAs, the electrodes are more closely spaced than in FMAs, but their maximum length of 1.5 mm limits the brain structures that are accessible. For example, the fingertip representation in area 3b is approximately 3 mm below the brain surface, along the posterior bank of the central sulcus (in macaques); this location is out of reach of UEAs yet can be accessed with FMAs, whose electrodes can be arbitrarily long. An electrode array typically impinges upon the cortical territory of two to four digits (often with some palm) in macaques, and upon one or two digits in humans (Fig. 7c). The density of the stimulation sites can be increased if individual electrodes include multiple contacts¹⁴¹, which could increase the bandwidth of ICMS-based sensory feedback (this remains to be tested).

Electrocorticographic electrodes placed just under the dura mater are an alternative to penetrating electrodes. Electrocorticographic electrodes do not penetrate neuronal tissue, and are therefore less invasive than electrodes for ICMS. However, the evoked sensations are more diffuse than those evoked by ICMS, and hence are often described as unnatural. They may also develop more slowly, taking several hundred milliseconds to be perceived¹⁴². Nonetheless, surface stimulation is sufficient to guide the performing of simple tasks¹⁴³, and promotes the embodiment of the prosthetic arm¹⁴⁴.

The sensory consequences of electrical stimulation of the somatosensory cortex.

In both monkeys and humans, ICMS delivered through a single electrode evokes sensations that are localized to a small patch of skin, such as a finger pad or palmar whorl^145,146. The location of the referred sensations is congruous with the receptive fields of the stimulated neurons. For instance, the electrical stimulation of neurons that respond to the index fingertip elicits a sensation referred to the index fingertip. This phenomenon suggests the existence of a ‘place code’ for perceived location; that is, the somatosensory cortex tracks where the body makes contact with objects by activating different parts of the body map. Hence, somatotopic mapping can also be used to intuitively convey information about the points of contact between an object and the bionic hand.

Sensitivity of the somatosensory cortex to ICMS has typically been measured using a two-alternative forced-choice procedure: users select which of two sequential stimulus intervals contains a faint pulse train, the amplitude of which varies from trial to trial over a certain range. The resulting behavioural performance is then described as a sigmoidal function of the pulse amplitude, and the threshold (which corresponds to 75% detection performance) is interpolated from this function. Detection thresholds (5–40 μA, with a median of about 20 μA^145–147) tend to be similar for humans and macaques. The thresholds decrease monotonically with pulse width, with stimulation frequency up to about 250 Hz and with pulse train duration until about 200 ms¹⁴⁷.

The perceived magnitude of a stimulus grows monotonically and nearly linearly with the stimulus amplitude (according to subjective reports¹⁴⁶). Sensitivity to changes in amplitude, as gauged by the just-noticeable difference, tends to be similar in humans and macaques (15–30 μA^145–147). Interestingly, just-noticeable differences are nearly independent of the amplitude range over which they are measured (in contrast to natural sensory stimulation and regardless of modality). This is in an apparent violation of Weber’s law, which states that the just-noticeable difference is proportional to the intensity of the reference stimulus¹⁴⁷. The violation of Weber’s law in artificial perception is attributable to the lower variability of the neuronal responses evoked by ICMS (compared with externally evoked stimulation). Whereas just-noticeable differences for ICMS amplitude are independent of pulse frequency, changes in ICMS frequency result in discriminable percepts, owing to changes in both intensity and quality^{139,140,148,149}.

The sensitivity to ICMS is well described by a model that simulates the responses of a volume of neurons whose response drops off with the square of distance from the electrode tip and is integrated over time¹⁵⁰. From this model, it can be inferred that the manner in which sensitivity to ICMS depends on the stimulation parameters is a straightforward product of known neuronal properties (such as refractoriness).

Humans often describe sensations evoked by ICMS as being natural or nearly natural^146,151. Although subjective ratings of quality are often unreliable and sample sizes are typically small, ICMS seems to evoke more naturalistic percepts than those of nerve stimulation. The quality of the percept varies from electrode to electrode, ranging from tingling and vibration to slow tapping and pressure^146,148,151. The quality evoked by the stimulation of a given electrode is unpredictable a priori, and varies with amplitude and frequency^148,151. Some users report limb movement, particularly at high ICMS amplitudes¹⁵¹, but it is unclear whether these reflect proprioceptive sensations per se or whether they reveal sensations of skin movement.

Artificial touch for guiding behaviour.

A main function of somatosensory feedback is to guide manual interactions with objects. Several models have been used to assess the degree to which ICMS evokes interpretable sensations that can be used to guide behaviour. In one study, monkeys used a brain-controlled virtual arm to explore their environment so as to locate different textures (each corresponding to a different temporal pattern of ICMS), and to select a ‘target’ texture corresponding to a specific ICMS pattern¹⁵². Following a similar example, a human with tetraplegia chose a virtual object target hidden inside a virtual ‘grab bag’ with other virtual objects, each corresponding to a different ICMS sequence¹⁵³. Rats, monkeys and humans can also use ICMS to navigate in two dimensions. In one task, infrared light sources were located by associating the ICMS of the somatosensory cortex to the output of infrared sensors fixed to the head of the rat¹⁵⁴ or to the brain-controlled bionic hand of an individual with tetraplegia¹⁵⁵. In another task, the monkey’s target location was cued via stimulation through different channels, which the animal learned to interpret over time¹⁵⁶. In an example of electrocorticography-based feedback, an individual adjusted the aperture of their hand to follow modulations in the intensity of electrical stimulation applied to their somatosensory cortex¹⁴³. In a demonstration of sensory feedback under more naturalistic conditions, an individual with tetraplegia controlled an anthropomorphic bionic arm through signals from the motor cortex while ICMS delivered to their somatosensory cortex tracked the output of torque sensors on the prosthetic fingers based on a somatotopic mapping¹⁵⁷. With this bidirectional brain–machine interface, and even in the presence of vision, the user achieved a significant improvement on a standard test of hand function (the ‘action research arm’ test) when using ICMS-based artificial tactile feedback.

Restoring proprioception via ICMS.

As is the case with peripheral nerve stimulation, the systematic elicitation of proprioceptive sensations via ICMS has been challenging. In one study, monkeys distinguished different patterns of ICMS delivered to the cortical field devoted to proprioception (area 3a; ref. ¹⁵⁸), but the nature of the evoked sensation remains unknown. In another study, ICMS delivered to area 2 was found to evoke a sensation of movement in the preferred direction of the stimulated proprioceptive neurons¹⁵⁹; however, this result could not be replicated. In a more compelling demonstration of artificial proprioception, a human user described ICMS percepts as proprioceptive¹⁵¹; yet these may also be described as tactile motion. Therefore, it remains to be determined whether ICMS can naturalistically convey information about limb state. A learning-based approach to artificial proprioception in which different stimulation channels steered movement to different directions within a planar workspace supported navigation in two dimensions^155,156. The users could use ICMS to guide reaching movements toward unseen targets. However, it is unclear whether this strategy scales to a more complex space of movements¹².

Biomimetic mappings.

Somatosensation can be restored via electrical stimulation by exploiting existing body representations. For example, the location of contact between the prosthesis and an object can be signalled by electrically stimulating the somatotopically appropriate cortical neurons. When the prosthetic fingertip touches the object, electrical current is delivered through an electrode that impinges upon the corresponding fingertip representation in the somatosensory cortex, thereby evoking a sensation referred to that fingertip. This mapping results in the intuitive transmission of information about contact location.

Similarly, the sense of touch conveys precise information about contact pressure, allowing us to exert just enough force on objects to pick them up. Without such pressure signals, we grasp objects with much more force than necessary⁹. In the somatosensory cortex, increases in pressure applied to the skin involve two neural correlates: neurons that are most sensitive to the stimulus (with receptive fields right under the stimulus) become more active, and nearby neurons (with nearby receptive fields) are recruited. A qualitatively analogous progression is achieved by increasing the amplitude of the stimulation pulses. Changes in contact pressure can thus be signalled intuitively by modulating the amplitude of ICMS, as has been demonstrated in monkeys¹⁴⁵ and humans¹⁴⁶. Although this strategy captures the time-averaged neural correlates of changes in pressure, it overlooks the dynamics of cortical responses during contact. In fact, the response of cortical neurons during contact onset and offset dwarfs the response during the maintenance of contact; this is because the somatosensory cortex responds preferentially to changes in contact pressure rather than to continuous contact pressure⁷. As discussed above, building these dynamics into sensory encoding algorithms improves the functionality of feedback conveyed through electrical stimulation of the peripheral nerve, and may also yield better ICMS-based feedback.

Biomimicry might also be used to convey other types of sensory information. For example, the electrical activation of motion-selective neurons may be used to convey information about the relative motion between the prosthesis and an object, and the electrical stimulation of orientation-selective neurons may be used to signal contact with an object’s edge¹³.

The role of learning in neuroprosthetics.

ICMS evokes highly unnatural patterns of neuronal activation. All activated neurons are entrained with the electrical pulses, which results in unnaturally synchronized activation across large neuronal populations. Also, ICMS produces prolonged and widespread neuronal depressions lasting 80 ms or longer¹⁶⁰; these are not observed during natural stimulation. Attempting biomimetic neuronal activation with ICMS is thus a fool’s errand. Rather, systematic (yet perhaps arbitrary) mappings between sensors on the prosthetic hand and ICMS can be created and learnt by users through chronic exposure via interactions with objects. Animals and humans can in fact learn to use new sensory mappings to perform tasks, such as finding targets hidden within the workspace^152,153, reaching toward hidden targets^155,156 and navigating¹⁵⁴. Hence, biomimicry may not be necessary to achieve useful sensory feedback.

However, it is unclear whether this approach can scale to reach the complexity of a human limb¹². The arm and hand can move in many different ways, and thousands of tactile channels convey information about contact events. The tactile space can be reduced considerably to a handful of channels that ‘tile’ the hand. Yet whether such a complex sensory space—in which each degree of freedom of movement and each contact event is tracked through a novel sensory channel—can be learned remains to be shown. Although the brain’s ability to learn is staggering, the massive reorganization required to achieve a new sensory representation de novo is likely to be impossible in adulthood. Indeed, sensory representations in the adult cortex are highly stable even after the massive de-afferentation involved in amputation⁹⁹. Nevertheless, given the obvious unnaturalness of ICMS-evoked neuronal activation, the nervous system will have to adapt to some degree to a new artificial sensory modality, and some degree of adaptation has been shown to be possible.

The future of bionic touch

Several challenges must be overcome before sensorized bionic hands can become clinically available. First, the benefits of sensory feedback have been demonstrated mostly under artificial laboratory conditions rather than in activities of daily living under ethological conditions. Still, recent studies have shown the benefits of artificial touch during prolonged home use of bionic hands by amputees^100,161. Such demonstrations are steps toward justifying the effort and costs of sensory restoration to spur clinical deployment.

Second, the relative strengths of the available technologies have not been sufficiently quantified. Quantitative comparisons are however complicated by the fact that different user populations (or even different users within a population) will differentially benefit from different technologies. For example, an individual with spinal cord injury will not benefit from peripheral nerve stimulation, and amputees may benefit to varying degrees from the different technologies, depending on the type of amputation (such as distal or proximal).

Third, it is not always clear whether invasive technologies confer sufficient additional benefits to supplant non-invasive approaches. In general, non-invasive technologies are limited by bulkiness, by the need for daily donning and doffing, and (in principle) by the bandwidth of the sensory feedback that they can confer. However, requirements for non-invasive approaches are less stringent than for invasive ones. As non-invasive approaches become easier to don and doff, are made more cost-effective and biocompatible for prolonged skin contact, and meet electrical safety requirements, they will become more attractive commercially. As implanted systems become safer, longer-lasting, more biocompatible and deliverable by accessible medical personnel, their clinical viability will improve.

Fourth, whether the naturalness of artificial touch is inextricably linked to its utility is questionable. Naturalness is not a pre-condition for usability, as evidenced by the fact that individuals can be trained to use unnatural sensory feedback. Although biomimetic artificial touch improves the functionality and embodiment of a bionic hand^{101,106,113,117,118}, the naturalness of an electrically evoked sensation is limited by the lack of naturalness in the evoked pattern of neuronal activation, which is characterized by synchronized spiking across groups of neurons. Such synchronized spiking is certain to impact naturalness, but does it significantly impact utility? Evidence from work with amputees suggests that if the envelope of the aggregate neuronal activity is sufficiently naturalistic, it will be interpretable by downstream structures—as was found to be the case with cochlear implants. As the qualitative experience and functional benefits of biomimetic stimulation regimes are measured in tandem, the relationship between quality and function will become clearer.

Current non-invasive approaches are based on mechanical stimulation or (transcutaneous) electrical stimulation. Yet ultrasound may provide an alternative means to activate neurons without the need for surgery¹⁶². For invasive neuroprosthetics, optogenetics^163–166 may allow for stimulation that is more targeted and selective; however, the technology has yet to be tested in humans. The widespread use of deep brain stimulation to alleviate the symptoms of Parkinson’s and of depression (and the accompanying development of such electrode technology¹⁶⁷) may enable sensory feedback via thalamic stimulation^132,133.

Developments in sensory feedback will spur the need to improve the sensorization of bionic hands. Current bionic hands comprise a small number of sensors (one or a few per digit). This impoverished sensorization severely constrains the informativeness of the feedback. Although sensors are currently not a bottleneck¹⁶⁸, future feedback interfaces may be more sophisticated and selective, and require sensors with greater spatiotemporal resolution. Moreover, sensorized artificial skins that mimic signals conveyed by native mechanoreceptors may offer promising alternatives to state-of-the-art bionic sensors^169–173. Ultimately, rich sensory feedback might allow amputees and individuals with tetraplegia to approach the dexterity of able-bodied individuals.