Restoring Tactile sensations via neural interfaces for real-time force-and-slippage closed-loop control of bionic hands

Abstract

Despite previous studies on the restoration of tactile sensation on the fingers and the hand, there are no examples of use of the routed sensory information to finely control the prosthesis hand in complex grasp and manipulation tasks. Here it is shown that force and slippage sensations can be elicited in an amputee subject by means of biologically-inspired slippage detection and encoding algorithms, supported by a stick-slip model of the performed grasp. A combination of cuff and intraneural electrodes was implanted for eleven weeks in a young woman with hand amputation, and was shown to provide close-to-natural force and slippage sensations, paramount for significantly improving the subject’s manipulative skills with the prosthesis. Evidence is provided about the improvement of the subject’s grasping and manipulation capabilities over time, thanks to neural feedback. The elicited tactile sensations enabled the successful fulfillment of fine grasp and manipulation tasks with increasing complexity. Grasp performance was quantitatively assessed by means of instrumented objects and a purposely developed metrics. Closed-loop control capabilities enabled by the neural feedback were compared to those achieved without feedback. Further, the work investigates whether the described amelioration of motor performance in dexterous tasks had as central neurophysiological correlates changes in motor cortex plasticity and whether such changes were of purely motor origin, or else the effect of a strong and persistent drive of the sensory feedback.

Introduction

Human dexterity and manipulation capabilities are enabled by the hand complex bio-mechanics, a sophisticated sensory system and a sensorimotor control loop based on a bidirectional communication with the brain. Sensory contribution is so important that in peripheral nerves, sensory fibers largely outnumber motor axons; this is extremely pronounced in forearm and hand nerves (1). This is why, in case of hand loss, any attempt to restore a physiological motor control of hand prostheses should go primarily through the restoration of sensory information allowing a proficient sensory-motor integration. The impossibility of current prostheses to provide the user with a pleasant and meaningful sensory feedback is one of the main reasons of the high percentage of prosthesis use abandonment (>30%) (2).

Up to now neuroprosthetics research has shown that: (i) The most robust and accurate way to extract upper-limb prosthetic user’s intention is by decoding muscle electrical activity (3)(4); (ii) It is possible to deliver close-to-natural sensations to the human brain, such as pressure, object stiffness and shape, and texture (5)(6)(7)(8)(9)(10) by electrically stimulating peripheral nerves (11)(6)(12); (iii) Stability over time of the evoked sensations has been demonstrated up to 24 months for cuff and FINE electrodes (12)(13) (iv) User satisfaction resulting from the receipt of sensory feedback and alleviation of phantom limb pain is significant (11); (v) Sensory feedback promotes a sense of ownership (i.e. embodiment) of the robotic limb (17)(3).

In affected subjects, amputation distorts cortical areas devoted to the control of the limb (14)(15). A consistent reversion of the amputation-induced aberrant cortical plasticity has been reported after the use of bionic prostheses enabling some kind of sensory feedback, (11), (16), (17). Indeed, a robotic hand controlled through intraneural electrodes improved motor cortical representation of the lost hand (11), EEG activation pattern during movement of the phantom hand (18), functional interhemispheric interaction (19) and the cortico-cortical functional connectivity (20), and made them more physiological.

Invasive stimulation of peripheral nerves, exploiting the natural pathways of communication between the hand and the brain, may represents a very promising strategy to achieve a close-to-natural feedback (11)(6)(12)(13). Nevertheless, closed-loop control of complex grasp and fine manipulation through prosthetic hands is still a core challenge.

This work shows that it is possible to recover sensory-motor integration through neural electrodes, and enable real-time closed-loop control of bionic hands in tasks of fine grasp and manipulation, thanks to the routed sensory information. Neural electrodes, implanted in a young woman with hand amputation, were shown to provide close-to-natural force and slippage sensations, paramount for significantly improving the subject’s manipulative skills with the prosthesis. As it happens with physiological Fast Adapting (FA) units, which are sensitive to high frequency vibrations (22), slippage has been detected through vibrations induced in the force signals recorded by sensors embedded in the prosthetic hand; hence, slippage information has been packaged into spatiotemporally discrete patterns, which integrate signals across skin area and time (23), and has been delivered to the subject by means of nerve electrical stimulation. A stick-slip model was used for deducing the slippage stimulation strategy. The model showed that slippage generates a relative movement of the grasped object on the skin of the fingers that are involved in grasping. Hence, electrical stimulation was delivered sequentially to two adjacent fingers participating in grasping in order to produce a sensation that integrated signals across the skin area.

Force and slippage information were translated into electrical stimuli; this allowed the patient to actively control the grasp stability, modulate the force level and hinder the object fall with a myoelectric control of the prosthesis.

Modification over time of the stimulation parameters and the evoked sensations went in parallel with the improvement of the subject’s grasping capabilities in tasks of increasing complexity up to dexterous manipulation, quantitatively assessed by means of instrumented objects and a purposely-developed metrics. Closed-loop control capabilities enabled by the neural feedback were compared with those achieved without feedback. Closed-loop force-and-slippage control was replicated with both a research robotic hand prototype and a commercial prosthesis, showing that performance was independent of the adopted prosthesis.

Moreover, in parallel with manipulative skills, a major role in driving brain reshaping accompanying the employment of bionic prosthesis is the achievement of a functional sensorimotor closed loop through an effective bidirectional communication in the human–machine interfacing (15). Hitherto, the reported changes in mapping and connectivity in favor of an amelioration of cortical hand processing were static pictures and were limited to either the motor or sensory domain. Here, in order to extract a more dynamic index able to predict amputees’ ability to recover, the propensity of the brain to undergo plastic changes induced by the prosthetic training has been evaluated through the response to plasticity-inducing protocols, and the relative weights of pure motor vs sensorimotor induction of plasticity has been explored in one amputee controlling bidirectionally-interfaced bionic prostheses.

Results

A stick-slip model for validating the proposed slippage detection and encoding approach

The biological plausibility of the encoding algorithm adopted to elicit slippage sensations was investigated by developing a stick-slip model (23) of multifingered grasps (Figure 1). The main purpose of the model was to in-depth analyse the slippage mechanism in healthy subjects and deduce the slippage stimulation strategy for the amputee patient. The model (described in detail in Methods) replicates realistic conditions of multifingered grasps of an object under the effect of gravity, and accounts for the load force due to the object mass (F_p) and the elastic force due to skin elasticity (F_e). F_s is the external disturbance that causes slippage. The input-output relationship allows determining the object displacement induced by the external force F_s, given the applied normal force, or alternatively the applied normal force, given the object displacement. Slippage causes a relative movement of the object on the fingers skin. The estimation of the object displacement by means of the model and the measurements on the healthy subjects allows evaluating the size of the skin area involved by slippage and, subsequently, allows defining the locations of the hand to be electrically stimulated.

Figure 1. Stick-slip model of a multifingered grasp.

(A) The model describes the mechanism of stick-slip during grasps involving from 2 up to 5 fingers. A tridigital grasp of an object is shown. F_n1 is the force applied by the thumb, while F_n is the resultant of the normal forces applied by all the fingers (i.e. F_n =F_n1 + F_n2 + Fn3). F_s is the external disturbance that causes slippage. When an external disturbance force F_s is applied to the spring, it will store elastic energy and an increasing force will be exerted on the object that is opposed by the frictional force F_t=F_t1 + F_t2 + F_t3. When F_t ≥ F_p − F_e + F_s the object sticks; on the other hand, when F_t < F_p − F_e + F_s the object slips. (B) Slip occurrence and corresponding force variation. (C) The object displacement caused by disturbance F_s and computed by the stick-slip model, and the displacement measured by the sensors on the object. The difference between the measured object displacement and the computed object displacement is not statistically significant (p=0.84).

The reliability of the model to predict the object displacement caused by perturbation F_s was proven by an on-purpose study on 10 healthy subjects performing power and precision grasps (see supplementary material). Normal forces and object displacement were measured to validate the model. Hence, given the measured normal force, the object displacement was also computed by means of Eq. (1). The study showed (Figure 1) that the object displacement computed by the model as 26.41±10.22 mm was comparable with the measured one, given by 28.12±8.95 mm (p=0.84, Wilcoxon Signed-Rank test). Moreover, very importantly, all the subjects referred that during slippage they felt the object flowing along the index and middle fingers because of the object displacement. This was also supported by the data, since the measured object displacement was comparable to the sum of the two finger sizes in the slippage direction, i.e. 32.20±4.76 mm. This achievement represented the key element of the adopted slippage stimulation strategy: i.e. slippage sensation was delivered with an electrical stimulation that flowed along the index and middle fingers in contact with the object.

Hence, for the amputee participant, slippage was detected through the induced vibrations that were found in the normal force signal (F_n) measured by the force sensors embedded in the prosthetic fingers, in a way similar to the afferent response in the natural hand (22). An ON/OFF signal was then generated by means of the online processing algorithm of the force signal described in (30)(32). Then, the slippage information was encoded as trains of cathodic rectangular biphasic electrical current pulses with fixed parameters sequentially injected on two adjacent fingers (i.e. index and middle fingers), in order to deliver a sensation that integrated signals across the skin area of the fingers in contact with the object.

Figure 2 shows the total force applied by the amputee participant in power and precision grasps performed with neural feedback; it was comparable to the total force obtained by the model adapted to the patient. In both cases slippage was hindered and the object was stably grasped. The displacement of the object on the prosthesis fingers induced by the slippage was given by 26.36±7.43 mm.

Figure 2. Grasp force from the model vs. the measured force from the amputee.

Closed-loop control with neural feedback in power (A) and precision (B) grasps. Sensors embedded in the object measured the normal component of the force (violet) and, after online processing, provided the slippage signal (red). The participant modulated the level of force, after feeling slippage through neural stimulation. Therefore, a stable grasp was achieved up to the end of the trial and the release of the object. The normal component of the force extracted from the model for the same perturbation condition is shown in light blue. All the traces are normalized with respect to the maximum forces exerted by the hand (i.e. 7.33 N for power grasp and 3.96 for precision grasp) and maximum time duration (i.e. 26.90 s for power grasp and 19.35s for precision grasp.

Real-time force-and-slippage closed-loop control

It was investigated the possibility to elicit in the participant close-to-natural sensations of grasping force and slippage through invasive nerve electrical stimulation (as described above), and use them to perform a real-time closed-loop control of force and slippage in tasks with increasing complexity (Figure 3).

Figure 3. Real-time force-and-slippage closed-loop control of hand prosthesis with neural feedback.

The sensory output produced by a biomechatronic hand embedding force sensors was routed back through neural stimulation to evoke close-to-natural force and slippage feedback. (A) Subject’s intention was decoded by the muscular activity, through surface EMG sensors in the socket and a pattern recognition algorithm that classified the gesture and the force level. (B) Position and force control was implemented on a biomechatronic prosthetic hand for performing the task. (C) The hand fingers with force-sensing resistors read the applied forces and detected slippage. (D) The measured force applied to the grasped object and the detected slippage event are encoded in force and slippage stimulation patterns. (E) Force and slippage sensations are delivered to the participant by means of cuff and intraneural electrodes. (F) Photograph of the surgical intervention for implanting cuff and intraneural electrodes in ulnar and median nerves

The blindfolded and acoustically shielded subject was asked to grasp objects placed close to the fingers of the prosthesis under two conditions: with neural feedback and without any feedback.

Four categories of tasks, ordered by increasing complexity, were tested: (A) Lateral grasp of large and small objects; (B) pick and place of large objects with a power grasp; (C) pick and place of small objects with a precision grasp; (D) manipulation tasks of pouring water from a bottle to a cup and shape sorter with small cylinders and discs. Complexity of the grasp was considered as related to the extension of the contact area between the object and the hand. Lower contact area, typical of precision and manipulation tasks, requires higher manipulative skills and more efficient control of grasp stability.

Force-sensitive-resistor sensors embedded in the prosthetic fingers measured the applied forces and provided a binary slippage signal, through the developed slippage detection algorithm presented in detail in (31)(32). Force and slippage feedback were provided to contacts number 10, 12 and 16 of the intraneural electrode in the median nerve that the subject referred to map on the thumb, index and middle fingers.

The two cases of use of neural feedback and absence of any kind of feedback are shown in Figure 4 and in Figure S7. When the object is touched, force feedback is provided. In the case of neural feedback the subject actively controls hand closing with the desired level of force, by producing a variation in the EMG signal related to the perceived force and slippage sensations. Instead, in the case of manipulation task without feedback, the object can fall because of slippage, forces vanishes accordingly, but the hand is still closed because of the absence of sensation.

Figure 4. Real-time force-and-slippage control of a manipulation task with neural feedback.

(A) With neural feedback. The participant performed a manipulation task of shape sorter of a small cylindrical object: the pinch gesture was selected by the EMG classifier and thumb and index fingers started moving. Once the object was touched, force feedback was provided. The slippage event was felt by the participant, who closed the hand and actively tuned the level of force by producing a variation in the EMG signal. Grasp stability was reached up to the end of the trial. Hence, the open hand gesture was classified and the hand re-opened. (B) Without feedback. The participant performed a manipulation task of shape sorter of a small cylindrical object: the pinch gesture was selected by the EMG classifier and thumb and index fingers stared moving. Once the object was touched, the applied force was measured and slippage was detected by the sensors. There was no stimulation. The patient was not able to feel the detected slippage event and, consequently, the object fell. The forces vanishes accordingly. At the end of the trial, the open hand gesture was classified and the hand re-opened.

Performance of closed-loop grasp control and improvement over time

The participant’s ability to grasp and manipulate objects with neural feedback and without feedback, and the improvement over time were monitored after the first week of training with the closed-loop control (i.e. week four, named T₀), in the middle of the training period (week seven, named T₁), and at the end of the experimental study (i.e. week ten, named T₂).

The participant was asked to perform twenty-four repetitions for each of the four aforementioned categories of tasks at each time point. The total number of trials was 96 for each observation period. Manipulative skills were assessed through the performance index named weighted success (defined in methods). One of the key point of this work is to show the subject’s capability to stably handle the object by actively managing forces and slippage, enabled by neural feedback. This means that slip events can occur, and the task is completely unsuccessful only when the object falls. Otherwise, the success measure is decreased by the slip occurrence.

The comparative analysis of the performance achieved with neural feedback and without any feedback showed that the difference was statistically significant for lateral (p = 0.0062) and power (p = 0.015) categories at T₀, for precision (p= 0.0045) and manipulation (p= 0.0009) categories at T₂ (Figure 5).

Figure 5. Temporal evolution of grasp performance without feedback and with neural feedback.

The participant’s grasp performance was measured through the weighed success and monitored over time. Four categories of tasks (lateral, power, precision and manipulation) were performed at T0, T1, and T2. Mean value and standard deviation of the weighted success index are shown for each time point. Statistical significance for the three time point is indicated with * (Friedman non-parametric tests, Wilcoxon post-hoc test, Bonferroni correction (p<0.016)). Statistical significance between neural feedback and no feedback is indicated with + (Wilcoxon Signed-Rank test, p<0.05).

In absence of feedback, the subject showed a relevant increase of performance with learning, probably due to the fact that she had never used a myoelectric prosthesis before this study. She started from very low performance at T₀, and improved over time. The difference was statistically significant for i) lateral grasp between T₀ and T₁ (p = 0.002) and between T₀ and T₂ (p = 0.002); ii) power grasp between T₀ and T₂ (p = 0.015); iii) manipulation tasks between T₀ and T₂ (p = 0.0078).

In case of real-time force-and-slippage closed-loop control with neural feedback, the subject achieved high performance already from the first time point, except for manipulation (which was more complex than the other tasks). For the manipulation category performance improved over time up to a value of 0.89 ± 0.08 at T₂, which was significantly different with respect to the value achieved at T₀ (p=0.015).

Dexterity of closed-loop control via neural feedback

It was also in-depth investigated the advantage of using neural feedback to improve dexterity at time T₂. The following addional performance indicators were considered: 1) the force index, which provides a measure of the capability to apply the appropriate level of forces to prevent the object from falling; 2) the execution time, which provides the speed of task accomplishment. A comparative analysis with the case of no feedback was performed in the four categories of tasks. Twenty-four repetitions per task category were performed. Furthermore, in order to assess the interoperability of the developed closed-loop control and the robustness of the achieved results, both a research prototype and a commercial prosthesis were adopted.

The participant achieved globally comparable performance with the two prosthetic hands Figure 6. The weighted success in absence of feedback was always lower than the success rate achieved with neural feedback (Figure 6). For precision and manipulation tasks the difference was statistically significant (p= 0.0045 and p=0.0009) for the research prototype; the difference was statistically significant for manipulation tasks (p=0.015) with the commercial hand. Similarly, the total applied force in case of no feedback was on average lower than the case of neural feedback (Figure 6). For the research prototype, the difference became significant for manipulation tasks (p = 0.0023), while for the commercial prosthesis the difference was significant for lateral grasp (p = 0.002), precision grasp (p = 0.02), and manipulation (p = 0.002). Moreover, the participant took in general more time to complete the tasks when no feedback was provided (Figure 6). The difference was statistically significant in manipulation tasks (for the research prototype: p= 0.006; for the commercial hand: p= 0.015) and lateral grasp performed with the commercial hand (p=0.019).

Figure 6. Grasp and dexterity assessment without feedback and with neural feedback and two different prosthetic hands.

The participant’s grasp performance and dexterity were measured through the weighed success, the force index and the execution time for the two cases of no feedback and neural feedback and two different prosthetic hands (a research prototype and a commercial hand). Statistical significance between neural feedback and no feedback is indicated with * (Wilcoxon Signed-Rank test, p<0.05). (A) Weighted success. (B) Force index. (C) Execution time. (D) Statistically significant differences between no feedback and neural feedback for the three indices and the two prosthetic hands. A significant improvement of grasp performance and dexterity are achieved in manipulation tasks, thanks to neural feedback, independently of the adopted prosthetic hand.

A user satisfaction questionnaire was administered to the participant at the end of the experimental study. She reported that the sensations elicited by the nerve electrical stimulation were evocative of the object slippage, very well distinguished from force sensation, and provided a quasi-realistic representation of the change of the contact area with the fingers involved in grasping (Movie S1).

Mapping of elicited sensations and change over time

Three electrodes elicited sensations in the patient as shown in Figure 7. Up to time T₀, the subject referred that most stimulations evoked a sense of movement (Table S1, Table S2, Table S3, Table S4). However, after T₀ and in particular when the subject began to extensively use the closed-loop control with neural feedback, the reported quality of sensations changed (Figure 7 B, Table S1, Table S2, Table S3, Table S4). Up to time T₀, 3 out of 16 contacts of the intraneural electrode in the median nerve and all the contacts of the cuff electrode evoked EMG responses. No EMG responses were obtained from all the other contacts. Contacts not evoking muscle twitch changed the induced sensation from movement to touch, and were used for the real-time closed-loop control.

Figure 7. Sensation locations and quality over time.

(A) The three electrodes elicited sensations in 13 different locations of the hand on anterior and posterior parts of the hand. Red areas refer to sensations evoked stimulating the ulnar nerve, while yellow represents territories elicited by stimulation on the median nerve. C1 indicates region elicited with cuff electrode on median nerve, C2 refers to cuff on ulnar nerve and I indicates the intraneural electrode in the median nerve. (B) Modification of the elicited sensations for the intraneural electrode on the median nerve. Up to time T₀ (i.e. PRE) most of the elicited sensations evoked movement (brown); after T₀ (i.e. POST) most of the elicited sensation evoked touch (blue). In separate series, histograms represent the cumulative percentage of stimulated contacts, considering all contacts, contacts evoking EMG activity (twitch +) and contacts evoking no EMG activity (twitch -).

Tests of sensorimotor integration

At T₀, the transcranial magnetic stimulation (TMS) test of short-latency somatosensory afferent inhibition (SAI) was very pronounced in a muscle involved in the amputation (i.e. flexor carpi ulnaris: 47%), while it was closer to a physiological value in a more proximal muscle, not involved in the amputation (i.e. biceps brachialis: 32%). The training period with sensory feedback induced a reduction of 16% in the overexpressed flexor carpi ulnaris SAI (from 47% to 31%), while left substantially unchanged biceps brachialis SAI (from 32% to 34%).

Tests of sensorimotor-induced cortical plasticity

Repetitive transcranial magnetic stimulation (rTMS) was used to test sensorimotor associative plasticity induced by inhibitory and facilitatory paired associative stimulation protocols (PAS- and PAS+, respectively), measured before T₀ (baseline) and after T₂ (post-training). The training with the sensorimotor closed loop bionic prosthesis induced a strong disinhibition of sensorimotor cortices, demonstrated by a consistent reduction of the effect of PAS- (from -47% to 0%) and by a still ineffective facilitatory PAS+ (from 3% to 3%) (Figure 8).

Figure 8. rTMS protocols.

Effects of rTMS protocols inducing changes in motor cortical excitability based on intracortical mechanisms (iTBS and cTBS, red bars) and on sensori-motor integration (PAS- and PAS+, blue bars), tested before T₀ (light bars) and after T₂ (dark bars). Values represent percent changes from baseline after each rTMS protocol. Dashed lines represent changes obtained with the same rTMS protocols in control subjects (data from (30)).

Tests of intra-motor cortical plasticity

RTMS was used to test intra-motor cortical plasticity induced by facilitatory (intermittent theta-burst stimulation, iTBS) and inhibitory (continuous theta-burst stimulation, cTBS) protocols, applied before T₀ (baseline) and after T₂ (post-training). At baseline, cTBS increased MEP amplitude by +66% and iTBS reduced MEP amplitude by 34%, showing opposite effects if compared to unimpaired subjects (30) (Figure 8). After training, cTBS produced only a slight facilitation (~+13%) and iTBS was no longer inhibitory but produced a slight facilitation. (~+4%) (Figure 8).

Discussion

Peripheral nerve electrodes implanted in the median and ulnar nerves were shown to recover sensory-motor integration in an amputee subject thanks to the provided force and slippage sensations, and enable manipulative skills through real-time closed-loop control of a bionic hand.

In the first phase of this study the stick-slip model extended to the multifingered grasp was used to predict object displacement during slippage. The results demonstrated that slippage causes a change in the contact area of the object with the fingers and that the object displacement involves two fingers, the index and middle finger in contact with the object. This laid the foundations of the stimulation strategy adopted for eliciting slippage sensations.

Afterwards, we verified whether the elicited tactile sensations could enable a physiological control of force and slippage during grasping and manipulation tasks. The subject actively controlled position and force of the prosthesis through voluntary muscle contractions. Once decided the type of task and position-controlled the prosthesis to accomplish the desired grasp, she was able to define the grasp force and autonomously increase or decrease it, thanks to the elicited sensations of force and slippage, and the online pattern recognition algorithm. The prosthetic hand was position controlled during hand preshaping; it was force controlled during grasping and manipulation. When force sensors on the prosthesis detected a slip event, a train of electrical stimuli on the fingers was delivered for the duration of the slippage. As a reaction the muscular activity produced a new pattern aimed at increasing the applied force level and preventing the object from falling. The time employed to measure the applied force and detect slippage via the algorithm in (31) (32) was under 50 ms. The hand controller acquired a new class from the classifier, including corrective actions due to slippage, every 100 ms. This delay is fully compatible with the time delays in human sensorimotor control loops engaged in corrective actions (~100 ms) (22) (24) (33).The overall closed-loop time for the sensorimotor control observed in the patient was around 500 ms. It is the time needed to decode the intended gesture, control the hand, provide the sensory feedback and apply a corrective action to avoid slippage.

On the other hand, when the participant was not provided with feedback, slippage was not felt, no muscular reaction was observed and the object fell, when not stably grasped. In particular, when the contact area between the hand and the object reduced (as in precision and manipulation tasks), grasp stability was more difficult to be ensured and the role of sensations became paramount (Figure 5). Hence, in absence of feedback the probability of object fall increased and, consequently, grasp performance decreased. All the tests were performed in absence of visual and auditory feedback, in order to avoid compensatory mechanisms due to other feedback modalities. It is expected that, in a real context of everyday life with visual and auditory feedback, the improvement of the subject’s grasp and manipulation capabilities enabled by the delivered force and slippage sensory feedback could be even more evident.

A weighted success index was introduced to assess the subject’s capability to stably handle the object. It was used to compare the neural feedback with no feedback, and to study the temporal evolution of manipulative skills. The results corroborate that sensory feedback needs time to be mastered. Figure 6 shows that grasp fundamental abilities were stable over time during the 11 weeks of experiments, while manipulative skills gradually increased and significantly improved by 25.7% at T₂. Performance was shown to improve with continuous usage as the subject learnt how to incorporate sensory feedback. Performance with neural feedback was always better than without feedback; the difference became statistically significant at T₂ for precision and manipulation tasks. For fundamental grasp categories (i.e. lateral and power grasps), performance improved over time also without feedback, probably because the subject had never used a myoelectric hand before the study.

Dexterity enabled by neural feedback was in-depth investigated by means of other two performance indicators: the force index and execution time. The comparative analysis at T₂ showed that neural feedback allowed achieving good manipulative skills. Performance achieved with neural feedback was always higher than no feedback. The subject performed all the task categories with higher forces and shorter execution time when neural feedback was provided. The difference became significant for manipulation tasks. This can be newly justified by the paramount role played by sensory feedback in more complex tasks. In the power grasp the object stability can be easily achieved also applying lower grasp force because of the wide contact area between the prosthetic hand and the object. This explains why power and lateral grasps can be successfully performed also when no feedback is provided. Precision and manipulation tasks are characterized by a reduction of the contact area between the object and the prosthetic hand. This entails a mastered control of the applied forces to ensure stability. It seems that the neural stimulation provides a rapid and effective sensory feedback which allows finely tuning the applied forces; in line with the spared mechano-transduction time, shortness of pathways and lower cognitive load. Indeed, the execution time is reduced accordingly.

All the results were confirmed by the tests performed also with a commercial hand. The participant was able to perform the four categories of tasks with similar performance and dexterity, thus proving interoperability of the system and robustness of the achieved results.

The sensorimotor closed loop training done by the participant induced a normalization of the quality of afferent stimuli, evolving from movement sensations to tactile sensations. This could be caused by a reeducation of central processing of the stimuli. In favor of a sensory driven amelioration of sensorimotor central processing there was also the very strong and consistent effect seen in the participant for sensorimotor induced plasticity, in line with the sensorimotor closed loop abilities of our system. In order to establish plasticity and relative weights of motor-motor drive or sensori-motor drive specific rTMS neuromodulatory interventions were exploited, mostly relying on different neurophysiologic mechanisms: theta-burst stimulation for the first, and paired associative stimulation for the latter (25)(26)(27). Plastic changes induced by rTMS protocols in the motor cortex contralateral to amputation were investigated before T₀ and after T₂. The training induced a reduction of facilitation of pure motor intervention (cTBS) and a reduction in the inhibition induced by sensorimotor driven plasticity (PAS-). Coherently, the level of cortical afferent inhibition measured with SAI testing was also reduced.

Overall, the observed effects indicated that intra-motor cortex plasticity became closer to the level observed in normal subjects (15) (8). However, the training did not induced a hyper-expressed intra-motor cortex plasticity, decreasing the likelihood of a pure motor origin of the achieved improvement of performance in our participant. On the contrary, inhibitory cortical phenomena related to integration of afferent information, expressed by either cortical afferent inhibition or cortical associative plasticity, are strongly reduced with sensory training. With the obvious caution due to the fact that our findings come from the analysis of a single case, the above data strongly supports that the increased efficacy of afferent information is the main factor responsible for favoring motor learning during the training.

Methods

Study design

In order to stimulate deep and surface nerve fibers, a combination of cuff and intraneural electrodes was implanted in the median and ulnar nerves of an amputee’s residuum for eleven weeks. The sensory output produced by a biomechatronic hand during grasping and manipulation tasks was relayed through neural stimulation to the participant to evoke real-time close-to-natural force and slippage feedback related to her missing hand.

To show the biological plausibility of the slippage encoding algorithm, a stick-slip model of multifingered grasps was developed and described below. The model was used to in-depth analyse the slippage mechanism in healthy subjects and deduce the slippage stimulation strategy for the amputee patient. Ten healthy subjects were recruited to validate the model (see supplementary materials). Based on these observations, the strategy for encoding slippage information by means of neural electrical stimulation in an amputee subject was defined.

In the experimental study with the amputee participant, closed-loop force-and-slippage control based on the elicited sensations was carried out in four categories of tasks with increasing complexity: A) Lateral grasp of large and small objects; (B) Pick and place of large objects with a power grasp; (C) Pick and place of small objects with a precision grasp; (D) Manipulation tasks of pouring water from a bottle to a cup and shape sorter with small cylinders and discs. The four categories involved:

• ▪Different fingers (from 2 to 5) in lateral, precision and power grasp configurations;
• ▪Grasping tasks (e.g. pick-and-place) as well as manipulation tasks (e.g. pouring, shape sorter);
• ▪Objects of different shapes (cylinder, parallelepiped, disk, cube, triangle), volume (2.54⸱10³ ÷ 2.65⸱10⁶ mm³) and weight (18.28 ÷ 198.65 g).

A comparative analysis with the case of no feedback was carried out. Twenty-four repetitions for each task category was performed in two different conditions: (i) no feedback (n=96 trials); (ii) force and slippage feedback through neural stimulation (n=96 trials). Moreover, improvement of grasping and manipulation capabilities was monitored over time at three time points (i.e. T₀, T₁ and T₂). Grasping performance was measured through three performance indicators.

The same set of trials were repeated at T₂ with two different biomechatronic prostheses (n=384 trials) in order to demonstrate that the results are general and independent of the employed prostheses. During the experiments, the participant was blindfolded and acoustically isolated. In this way, with neither vision nor auditory input, compensatory mechanisms due to other feedback modalities were avoided, and the improvement of sensory-motor performance entailed only by the delivered force and slippage feedback was assessed.

Finally, a neurophysiological assessment was carried out. It investigated if the described amelioration of motor performance in dexterous tasks had as central neurophysiological correlates changes in motor cortex plasticity and if such changes were more likely of purely motor origin, thus relaying on intra-motor cortex activity, or rather the effect of a strong and persistent drive of the sensory feedback.

Subject recruitment

This study was conducted at Campus Bio-Medico University Hospital of Rome in accordance with the Declaration of Helsinki and following amendments, and was approved by the local Ethics Committee and the assigned office of the Italian Ministry of Health. The volunteer subject signed an informed consent form.

The enrolled subject was exposed to an explosion that produced a trans-radial left upper limb amputation almost 30 years before. She is a right-handed female, 40 years old at the time of the experiment. She demonstrated good intellective abilities and comprehension.

The surgical procedure for implanting neural electrodes is described in supplementary materials.

Experimental setup

The bionic system was composed of commercial devices and research prototypes. Two biomechatronic hands were used, i.e. the IH2 Azzurra (Prensilia s.r.l.) and the RoboLimb (TouchBionics s.r.l.). They were equipped with force-sensing resistors (Interlink Electronics Inc.) for measuring normal forces between fingers and objects, and detecting slippage, thanks to the algorithm in (30). This detects the vibrations in the force signal due to sliding movements. A custom made socket was developed by Inail Prosthetic Center.

The tactile sensation was restored electrically by stimulating median and ulnar nerves via cuff and intraneural electrodes (ds-FILEs). The ds-FILE is characterized by 16 active contacts and 2 ground electrodes arranged on both sides of the structure (29). The wrap cuff electrode (Ardiem Medical, Inc.) is made of a total of 14 active contacts and 2 ground channels, distributed on 4 rings. The Multichannel System STG4008 stimulator was connected to the electrodes thanks to a custom-developed hub facilitating the channel selection.

For decoding muscular activity, six commercial active surface EMG sensors (Ottobock 13E200 = 50, 27 mm x 18 mm x 9,5 mm) were embedded into the socket. The data was sampled at 1 kHz frequency and with 12 bits resolution. The subject was instructed to reproduce with her phantom limb one of five gestures, i.e. “Rest” (relaxed hand), “Power” (hand with all fingers closed), “Pinch” (hand precision grasp with two fingers), “Open” (hand with all fingers opened), and “Lateral” (hand lateral configuration), and three levels of forces, i.e. high, medium and low. Once classified the level of force for a given gesture, the corresponding grasping force was applied by the prosthetic hand. Force-sensing resistors embedded in the prosthetic fingers measured the applied force and checked slippage. If slippage was detected, a slippage sensation was delivered to the subject, who could apply a corrective action. To this purpose, the grasping force was proportionally varied with the EMG signal, in order to promptly oppose slippage and prevent the object from falling. The raw filtered sEMG signals were taken as input features to a pattern recognition algorithm based on a Non-linear Logistic Regression (NLR) algorithm (Fig. S6), as described in (34). The classifier ran on an embedded system that relied on an ARM4 32bit NXP microcontroller with a 128Kb flash memory and 100MHz clock frequency.

Stick-slip model of multifingered grasp

The proposed model is an extension of the stick-slip model in the literature to the more realistic situation of a multifingered grasp of an object under gravity conditions. In Figure 1, F_n1 is the force applied by the thumb, while F_n is the resultant of the normal forces applied by all the fingers (i.e. F_n=F_n1+ F_n2+ F_n3 in Figure 1). Force tangential components F_t1, F_t2 and F_t3 are related to the normal components through the coefficient of kinetic friction µ. F_p accounts for the load force, F_e is the elastic force generated by the skin elasticity, and F_s is the external disturbance that causes slippage; it is modeled as a step function (F_s = Fu(t)). Skin elasticity is modeled with a spring.

When a force F_s is applied to the spring, it will store elastic energy and an increasing force will be exerted on the object that is opposed by the frictional force F_t = F_t1 + F_t2 + F_t3.

When F_t ≥ F_p − F_e + F_s the object sticks; on the other hand, when F_t < F_p − F_e + F_s the object slips. The equilibrium can be written as

$$F_t=F_p-F_e+m\ddot{x}-F_s$$

being m the mass of the object and $\ddot{x}$ the object acceleration. The object displacement induced by the external force F_s can be computed as

$$x\left(t\right)=\frac{a_0}{{\omega }_n^2}\left(u(t)-\text{cos}\left({\omega }_{n}t\right)\right)$$ (1)

where

$${\omega }_n^2=\frac{k}{m};a_0=\frac{1}{m}(\mu F_n-mg-F)$$

and k is the skin stiffness.

Electrical stimulation for sensory feedback

Force sensation was elicited by means of a train of 3 cathodic rectangular biphasic pulses with fixed frequency of 50 Hz and a fixed pulse width of 80 µs. The current amplitude was directly proportional to the voltage provided by the force sensors as

$$\frac{V-V^{low}}{V^{high}-V^{low}}=\frac{I-I^{low}}{I^{high}-I^{low}}$$

where V was the readout of the FSR, I was the current amplitude, I^low and I^high were the lowest and the highest current values tolerable by the subject, and V^low and V^high were the corresponding voltage outputs of the sensors (i.e. -4.15V and -3.5V).

Slippage sensations were elicited through a train of 3 cathodic rectangular biphasic pulses with fixed current amplitude (150 µA), frequency (50 Hz) and pulse width (80 µs). The slippage information was encoded as sequences of trains of three cathodic rectangular biphasic electrical current pulses with fixed parameters sequentially injected on index and middle fingers for the duration of the slippage event.

Grasp assessment

During the closed-loop control, sensors embedded into the hand fingers were used to measure force and slippage and produce a feedback signal for the amputee subject. On the other hand, objects were instrumented with force-sensing resistors for grasp assessment. Three performance indicators, named weighted success, force index, and execution time, were introduced.

The weighted success is a normalized measure of the task success rate and is expressed as the task success modulated by the number of occurred slippage events and normalized over the maximum number of slip events detected with the same feedback condition (see supplementary materials). It ranges in the interval [0, 1], where 0 is a failed trial and 1 is a successfully trial with no slippage.

The force index, expressed in Newton, measures the total force applied by the fingers involved in the grasping or manipulation task.

The execution time is the time employed for performing the task.

Neurophysiological assessment

Tests of sensorimotor integration

The effect of sensory afferent stimulation on motor cortical excitability was tested by combining electrical nerve stimulation with TMS of M1. The short latency afferent inhibition (SAI) paradigm was used (35), in which an electrical stimulus of the ulnar nerve, delivered immediately above the elbow, was followed by a TMS pulse over M1 at an interval exceeding of 2-4 ms the latency of the cortical somatosensory potential (SEP) evoked by stimulation of the same nerve.

For the test condition (M1 TMS alone) and for the three SAI conditions (ulnar nerve stimulus, interval, TMS where interstimulus interval lasted N20 latency +2, 3 and 4 ms), ten motor evoked potentials (MEP) were collected and averaged from both flexor carpi ulnaris and biceps brachialis muscles. For each interstimulus interval, SAI was then expressed as the percentage of reduction of the average MEP compared to the average MEP evoked by the test condition.

Tests of motor cortical plasticity

Motor cortical plasticity was assessed by means of repetitive TMS (rTMS) protocols at two time points; before T₀ and after T₂. Motor cortical excitability was assessed before and immediately after each rTMS protocol by single pulse TMS, as described above. Motor evoked potentials to 15 magnetic stimuli of M1 for each condition were collected and averaged from the biceps brachialis muscle.

RTMS was applied to the right primary motor cortex (contralateral to the amputated limb) using a DuoMAG XT-100 magnetic stimulator (Deymed, Czech Republic), producing a biphasic magnetic pulse. Four different rTMS protocols were used, based on the two general paradigms of theta-burst stimulation (TBS) (36) and of paired associative stimulation (PAS) (37): 1) the continuous theta-burst stimulation (cTBS), in which 3 pulses of stimulation were given at 50 Hz, repeated every 200 ms, for a total of 600 pulses; 2) the intermittent TBS (iTBS), in which 3 pulses of stimulation were given at 50 Hz, repeated every 200 ms and with a pause of 8 s every 2 s of stimulation, for a total of 600 pulses; 3) PAS- in which 90 pairs were delivered where each pair was made of a peripheral nerve electrical stimulation followed by TMS of the motor cortex with an interstimulus interval of 10 ms shorter than the latency of the cortical somatosensory evoked potential (ISI=10); 4) PAS+ in which 90 pairs were delivered where each pair was made of a peripheral nerve electrical stimulation followed by TMS of the motor cortex with an interstimulus interval exceeding of 5 ms the latency of the cortical somatosensory evoked potential (ISI=25). Such different rTMS-based protocols were chosen to explore, in a selective way, alternative possible mechanisms that result in cortical plasticity. Indeed, the changes of cortical excitability that can be seen after TBS protocols are due to the induction of synaptic plasticity in synapses between neurons all located within the motor cortex, thus TBS-induced changes of MEP can be considered as a proxy of pure motor cortex plasticity. Alternatively, repetitive stimulation of PAS protocols produces changes in cortical excitability due to synaptic plasticity in synapses between sensory and motor neurons, thus they can be considered as a proxy of the part of motor cortex plasticity driven by the sensory system, or in other words, sensori-motor associative plasticity.

One Sentence Summary.

The sensory output produced by a biomechatronic hand during grasping and manipulation tasks was relayed through neural stimulation to the participant to evoke real-time close-to-natural force and slippage feedback related to her missing hand.