Illusory movement perception improves motor control for prosthetic hands

Abstract

To effortlessly complete an intentional movement, the brain needs feedback from the body regarding the movement’s progress. This largely non-conscious kinesthetic sense helps the brain to learn relationships between motor commands and outcomes to correct movement errors. Prosthetic systems for restoring function have predominantly focused on controlling motorized joint movement. Without the kinesthetic sense, however, these devices do not become intuitively controllable. Here we report a method for endowing human amputees with a kinesthetic perception of dexterous robotic hands. Vibrating the muscles used for prosthetic control via a neural-machine interface produced the illusory perception of complex grip movements. Within minutes, three amputees integrated this kinesthetic feedback and improved movement control. Combining intent, kinesthesia, and vision instilled participants with a sense of agency over the robotic movements. This feedback approach for closed-loop control opens a pathway to seamless integration of minds and machines.

Introduction

Spatial awareness of the body, a cornerstone of purposeful movement, is informed by a host of sensory inputs from muscles, tendons, joints, and skin. During motor execution, this sense is used by the brain’s internal forward model to predict the physical outcomes of motor commands(1). These predictions are then compared to feedback coming from muscle motion sensors (the kinesthetic sense) to compute movement errors and make fine adjustments(1). Despite its central role in movement, how the kinesthetic sense operates is poorly understood, and currently there are no effective treatments to restore impaired kinesthesia(2).

Individuals with amputation are provided with increasingly sophisticated prosthetic options for restoring the lost ability to move, but less emphasis has been placed on restoring the lost kinesthetic sensation that guides movements. Many upper limb amputees still prefer older cable-actuated split-hook grippers and body-powered elbows because joint movements can be intuitively felt through the cable system(3, 4). By contrast, motorized prosthetic hands provide no meaningful feedback for movement or touch and must be carefully watched during the entirety of movement to perform even the simplest of tasks, much to the detriment of efficient control or multi-tasking(4, 5). In addition, vision is a poor substitute for kinesthesia(6) and cannot fully compensate for the loss of the intrinsic sensory mechanisms for movement prediction, error correction, experiential learning, and self-reference(7).

Substantial advances have been made in cutaneous touch feedback (pressure, tapping, moving touch, vibration, texture) in humans through the use of implanted and regenerative neural interfaces(8–12). However, kinesthesia (pure movement sensation) is an entirely distinct sensory modality from touch (contact and force sensation). Brain cortical electrical microstimulation, external feedback substitution, and learning approaches in a primate model showed that hand position information is critical for improving motor control(13). However, substituting external feedback for the lost intrinsic sensation requires learning new associations between these previously unrelated pathways. In human amputees, stimulation via implanted peripheral nerve electrodes can provide the ability to sense passive elbow joint positions and finger movements(12, 14), and movement feedback in closed-loop motor control was feasible and effective in functional single participant and single percept demonstrations of concept(15, 16). However, a comprehensive functional and perceptual framework for kinesthesia in applied prosthetic systems is still lacking(2, 13, 17, 18).

Beyond the critical role of kinesthesia in motor execution and error correction, the congruency between intentional movements and the immediate sensory feedback from the movements themselves provides a sense of authorship (agency) that distinguishes one’s own actions from those of others(19). Prosthetic limbs do not provide the movement feedback that makes the user feel as though he or she is in control of their actions. Establishing a sense of agency for these devices will help amputees intrinsically feel in control of their artificial limbs, a key aspect of user acceptance.

Here we used vibration-induced perceptual movement illusions in a human bidirectional neural-machine interface to generate the kinesthetic sensations of complex synergistic grip movements. This engineered perception of movement runs automatically within the amputee user’s intrinsic motor-control loop to improve real-time function without vision or other feedback, fuses intent and action through agency, directly applies to prosthetic hands, and is immediately translatable to a clinical prosthetic system.

Results

Vibration-induced kinesthetic illusions produce perception of complex grip movements in human amputees

In able-bodied individuals, vibrating limb tendons at 70 to 115 Hz generates a joint-specific perception of movement, even though the joint crossed by the tendon is not physically changing position(20). This kinesthetic illusion is strong enough to give an individual the impression that their extremities are assuming impossible positions(21, 22), and complex 3-dimensional arm movements have been simulated by inducing multi-joint kinesthetic illusions(23).

We investigated the use of vibration to elicit illusions of movement in a group of individuals with upper limb amputation who had undergone targeted reinnervation to create a biological neural-machine interface for prosthetic control and feedback(24–26). Specifically, motor and sensory nerves remaining after amputation were surgically redirected to reinnervate new proximal muscle and skin sites. Therefore, attempts to move the missing limbs, trigger contraction of the reinnervated muscles. The resulting electromyographic (EMG) signals are then used to intuitively control computerized motor-driven prosthetic limbs(25). A touch to the reinnervated skin feels like a touch to the hand that is no longer there, with normal sensory thresholds for vibration, pressure, heat, cold, and pain(27).

Using a hand-held vibration unit, we vibrated the proximal reinnervated residual muscles of 6 amputee participants (Par1-6; 90 Hz, 500 μm neutral to peak(28, 29)) and they used their intact hand to demonstrate what they felt(30, 31). Despite the complete absence of the distal limb from above the elbow or higher, every amputee participant spontaneously reported perceiving functionally relevant complex movement in their missing hand, wrist, or elbow (Fig. 1). We isolated 22 individual movement percepts across the 6 amputees. Muscles reinnervated by the median nerve provided various percepts of digit flexion. Muscles reinnervated by the radial nerve provided percepts of extension. The perceived movements were experienced as synergistic hand gestures despite being elicited in the biceps, triceps, brachialis, and pectoralis (Fig. S1), without cutaneous tactile correspondence in the overlying skin (Fig. S1). These results suggest that the sensory-neural structure of the elbow and shoulder muscles was reassigned through reinnervation by the nerves originally serving the hand. In 5 participants, we investigated the relationship between vibrational frequency and amplitude, and the reported magnitude of the vibration-induced illusion. Using a linear motor we applied vibration to the strongest reinnervated muscle percept site for each participant at 100, 300, or 500 μm (neutral-peak) displacements at 10, 30, 50, 70, 90, or 110 Hz. We found that the participants reported the highest magnitude estimates of illusory movement strength (0-5 on a Likert scale) for each of the three amplitudes at frequencies between 70 and 110 Hz (Fig. S2), which falls within the reported band of the kinesthetic illusion(20, 32, 33).

Fig. 1. Movement percepts for all participants.

Schematics representing perceived movements induced by 90 Hz vibration to the reinnervated residual muscles in 6 amputee participants (Par1-6) reported using the intact hand. Participant details are shown in Fig. S1. From a start position (grey outlines, Start), participants perceived movement in the direction and relative magnitude indicated by orange arrows to an end position (black outlines, Finish). Digits are specified by D1=thumb, D2=index finger, D3=middle finger, D4=ring finger, D5=little finger.

Active muscle contraction can influence either the amplitude(34) or the speed(35, 36) of the illusion, or abolish it altogether(37). We tested both active and passive configurations using physical and virtual prosthetic limbs substituting for the absence of an intact limb in our amputee participants. In conventional studies, active corresponds to the participant moving their own limb, whereas passive corresponds to the tester moving the limb for them (Fig. S3). In three amputees (Par 1, Par 2 and Par 5), we used a custom wearable vibration unit (kinesthetic tactor) to stimulate the muscles and a 22-sensor data glove worn on their remaining hand to capture the reproduced hand kinematics (angle, range, position, and speed) as they matched the kinesthetic illusory sensation from their reinnervated muscle in response to 90 Hz vibration under active and passive conditions(30, 31). Figure 2 shows representative finger joint kinematics for each participant’s hand-close percept that all progress from open to closed under different active and passive conditions on different days of the experiments (Fig. 2A). The illusory movement percepts had definitive stop positions, where the perceived hand motion ended even with continued vibration or muscle contraction (Movie S1). For two of the three amputees (Par 2 and Par 5), active movements were perceived as faster than passive ones (Bonferroni-corrected t-tests, Par 2 and Par 5 each p<0.001; Movie S1). The root-mean-square differences between the joint angle trajectories (Fig. 2B) and Pearson’s correlations (Fig. 2C) demonstrate that overall grip shape and involved digits were similar between the two conditions across all testing days for all participants. Furthermore, the targeted reinnervation amputees reported feeling the sensation of their hand closing when they contracted their muscle control sites. We refer to this as the intrinsic motor percept, and the root-mean-square differences between the joint angle trajectories and Pearson’s correlations from the data glove kinematics in Par 1 and 5 verified that vibration and active muscle contraction both elicited similar percepts (Fig. 2).

Fig. 2. Active, passive and intrinsic movement percepts with measures of similarity across days and percept types.

(A) Kinematic trajectories for the cylinder grip percepts for Par 1, Par 2, and Par 5 with the start (open) and end (closed) positions of the percepts demonstrated using the virtual hand at the top of each column. Graphs of digit average joint angles (n=30 trials [40 for Par 2 passive, first day]) are ranked in descending order according to average change in angle across all percepts and all participants. Individual plots show the joint with the greatest change in angle for that digit (PIP = proximal interphalangeal; MCP = metacarpophalangeal; IP = interphalangeal; Op. = opposition). Plots include active (teal), passive (grey) and intrinsic (magenta) percepts measured on the first experimental day (solid line), last experimental day (dashed line), and after the first speed game (dotted line, ASG1, see: Fig. S8B,C). (B) Aggregate measures of similarity in grip dynamics between each percept pair for each individual participant quantify percept stability across days and similarity between active, passive, and intrinsic conditions. The darker the shade, the greater the similarity between average percept joint trajectories (root-mean-square differences [RMSD] averaged across the joints in each digit with the greatest change in angle [n=5]). (C) Overall movement similarity by participant for all percepts across all days to quantify global percept stability. The farther the marker to the right, the greater the average correlation (Pearson correlation coefficients averaged across all of an individual’s percepts for the joints in each digit with the greatest change in angle [top, n=5 joints] and all joint movements [bottom, n=22 movements]).

Real-time illusory feedback improves movement control when using a virtual reality prosthetic system

To make efficient reach and grasp movements, people need to be able to control their grip aperture without looking at their hand. Normally individuals fixate on the object of interest, and then use proprioceptive feedback to preposition their hand(38). In contrast, motorized prosthetic users rely entirely on vision to position their artificial hands(39). To determine if kinesthetic illusory input could be used for natural prepositioning without vision, we linked the EMG hand-close signal (cylinder grip) in Par 1, Par 2 and Par 5 to the movement of a virtual prosthesis rendered with the MuJoCo HAPTIX physics engine (see: methods). Closing the virtual prosthetic hand triggered the control system to elicit the illusory percept of cylinder grip movement by vibrating their reinnervated biceps with a wearable kinesthetic tactor. We randomly showed the participants images of their own percept at 25, 50, 75, and 100% closed positions and instructed them to close their hand to match the position of the picture. Each time they closed the virtual prosthesis, they were randomly provided with 90 Hz illusory kinesthetic feedback, 20 Hz sham vibratory feedback, or no vibratory feedback (as a normal motorized prosthetic hand would operate) (Fig. S4A). They could not see their virtual prosthesis, from which we recorded grip kinematics, and they were not informed of their performance during the testing. For ideal proportional performance participants should have equal time to close intervals for each target step (Fig. 3A, black rectangles). We found that with vibration-induced kinesthetic illusory feedback the amputee participants did not need to see their hands to control them accurately. With 90Hz stimulation all participants achieved near ideal proportionality of close time intervals as can be seen in the close registration between the black and teal rectangles in Figure 3A. Whereas, with 20 Hz sham (purple rectangles) and no feedback (orange rectangles), the participants showed less registration with the ideal proportional goals. Although this effect was most noticeable in Par 2, who had little sense of hand position without vibratory feedback (orange rectangles), the illusory input also improved the performance of Par 1 and Par 5 who at baseline already operated their devices well with respect to this task. Furthermore, with the 90Hz stimulation feedback all of the participants performed indistinguishably from a cohort of 5 able-bodied individuals presented with an analogous task (Fig. S4B). The amputees’ values for degree of alignment with the ideal proportional performance (dashed lines shown in Fig. 3A) clustered together within two standard deviations of that for the average able-bodied participants (Fig. 3B). Alignment values were 0.014, 0.024, and 0.032 for Par 1, Par 2, and Par 5 respectively and 0.064 ± 0.029 for able-bodied participants (alignment is the root-mean-square difference between actual and ideal time to target normalized to the ideal time to a target close grip of 100%).

Fig. 3. Performance in functional tasks with and without vibration-induced kinesthetic illusory feedback.

(A) Three participants’ (Par 1, Par 2, and Par 5) ability to accurately reach proportioned intervals in a grip-conformation task (25%, 50%, 75%, 100% hand closed) while receiving no (0 Hz, orange), 20 Hz (purple), and 90 Hz (teal) vibratory feedback. Actual intervals between targets (colored rectangles) are compared to the ideal intervals fit to each participant’s actual performance times (black open rectangles). The actual times to target position are shown as circles with error bars indicating 95% confidence intervals (n=20), which specify the height of the colored rectangles. The ideal change in time to target position between percent close positions (black open rectangles) is specified by the linear regression with intercept set at zero (black dotted line). Alignment between the black open rectangles and the colored rectangles is an indication of the participant’s ability to reach the proportional degrees of closure. (B) Line graph showing degree of alignment with ideal proportional performance in the grip-conformation task shown in panel A for amputee participants (Par 1, Par 2, and Par 5) and in an analogous task for an able-bodied cohort (AB Avg, n=5, Fig. S4B). The black dashed line indicates the average performance of able-bodied (AB Avg) participants ± 2 standard deviations (grey shaded area). (C) Bar graph showing average adaptation rate to self-generated error for Par 1 and Par 2 in different feedback conditions (vis.+kin. = vision and kinesthesia; kin. only = kinesthesia only; vis. only = vision only; sham only = 20 Hz vibration). Error bars represent 95% confidence intervals (n=75-95 trials). (D) Bar graph showing the standard deviation of the overall system noise (see methods for details) for Par 1, Par 2, and Par 5, for different feedback combinations (vis.+kin. = vision and kinesthesia; kin. only = kinesthesia only; vis. only = vision only; sham only = 20 Hz vibration). (E) Graph of cumulative EMG control signal trajectories for Par 5 using an agonist-antagonist muscle pair (biceps, hand close; triceps, hand open). Average cumulative control signal trajectories (n=4 trials, time the participant provided a close signal [negative] plus the open signal [positive]) for each feedback condition (90 Hz [teal line], 20 Hz [purple line], or no vibration [orange line]) compared to the target trajectory (black line).

When correcting for errors in movement, the degree of response takes into account the uncertainty intrinsic to each type of feedback characterizing the movement itself (40, 41). Feedback with higher fidelity (less uncertainty) is trusted more. The magnitude of error correction, quantified as a trial-by-trial adaptation rate, lets us assess the trust afforded to the feedback provided relative to the participant’s expectation (internal model) of their intended movement(42, 43). Increased trust in feedback leads to an increased trial-by-trial adaptation rate when calculated from the participant’s correction to self-generated errors without external perturbation(40). We used the trial-by-trial adaptation rate to measure the level of trust afforded to the kinesthetic illusory feedback for Par 1 and Par 2. The participants completed blocks of repeated grasping trials under four specific feedback conditions (visual+90 Hz kinesthetic, 90 Hz kinesthetic alone, visual alone, or 20 Hz sham alone; Fig. S5A). They used a virtual prosthesis to catch a falling block at a pre-determined position while we recorded their movement errors (Fig. S5B). After a brief familiarization, they caught the block 25 consecutive times for each feedback condition, totaling 100 trials before a break. This was repeated 5 times, with the presentation order of each condition randomized each time. We calculated the trial-by-trial adaptation rate as a slope of the magnitude of correction from one trial to the next, relative to the magnitude of error for each trial itself (Fig. 3C, S5C). Unrestricted vision is a trusted source of feedback(41, 44). We found that in both Par 1 and Par 2 kinesthetic illusory feedback alone appeared to have adaptation rates in a similar range to those observed for vision only and vision+kinesthesia (Fig. 3C). Sham vibration appeared to tend towards lower adaptation rates, corresponding to lower feedback trust (Fig. 3C). Although statistical significance could not be established due to low subject numbers, we observed these trends in both participants. Adaptation rates for sham feedback were 19.1-28.4% lower than the rates for the other feedback types. This suggests a decrease in adaptation rate that aligns with differences observed between treatment groups in other studies (14.5-20.5% differences in adaptation rate(42)).

In Par 1, Par 2, and Par 5 we compared the relative importance of vision to vibration-induced illusory kinesthesia on task performance, to see if the two channels could be optimally integrated, as shown previously with the combination of visual and haptic feedback in able-bodied humans(45). We calculated system variance estimations representing the combination of sensory and controller noise using a two-alternative forced choice task. Participants used the EMG controlled virtual prosthesis to catch a falling block at a specific position and were asked to choose which of two grasping trials was perturbed by a random externally-applied control onset delay (Fig. S5D). The control delay was decreased for correct selections and increased for incorrect selections resulting in an adaptive staircase that was used to adjust the perturbation-delay magnitude to establish the 84% level just-noticeable-difference(46) (Fig. S5E). Limited subject numbers prevented robust statistical analysis but consistent trends emerged. With the four feedback conditions, we found that kinesthesia always provided lower uncertainty than vision, even for the participants who fixated on vision and seemed unaware of the vibrations. Sham vibration feedback resulted in high variability of system variance across the three participants whereas the other feedback modalities were stable. Furthermore, all participants also integrated vision and kinesthesia to form an even more accurate estimate of their system’s performance (a lower JND). Ernst and Banks’ optimal estimation integration equation(45) mathematically predicts the optimal fusion for such systems (see: methods) – Par 1 and Par 2 operated within milliseconds of these predictions (Fig. 3D). These results suggest that the vibration-induced illusory kinesthetic perception provides lower feedback variance than the other modalities provided, and it is nearly optimally integrated together with vision when movement sense is coupled to motor control.

Simultaneous vibration of agonist-antagonist muscle pairs abolishes the illusory perception of limb movement(20). We next determined if the kinesthetic illusory approach would work in the context of a prosthetic system, where agonist-antagonist muscle pairs are used to control opposing hand movements. Par 5 was fit with two sets of EMG electrode/tactor (movement-control/feedback-perception) pairs, running in parallel, on the reinnervated biceps (hand-closed coupled to the cylinder grip percept) and triceps (hand-open coupled to the hand-open percept) (Fig. S6). We provided three feedback conditions (90 Hz illusory kinesthetic vibration, 20 Hz vibration, and no vibration [0 Hz]) and, as with a normal prosthesis, contracting both muscle control sites simultaneously (co-contraction) did not produce movement of the virtual hand and did not trigger a feedback condition. We showed the participant a video of a virtual prosthetic hand repeatedly performing symmetrical cycles of close-open-close movement that they watched and attempted to track with their EMG-controlled virtual prosthesis, which was hidden from view (Movie S2). We compared the time spent closing and the time spent opening the virtual hand to those in the target video. We found that with 90 Hz kinesthetic feedback (teal line Fig. 3E) Par 5′s close-open-close signal cycles most closely followed the target signal (black line Fig. 3E). While less time was spent closing, the time spent opening was not significantly different than the target (Bonferroni-corrected chi-square goodness-of-fit test, χ²[2, n = 2004] = 173.3, p < 0.001; Bonferroni-corrected z-tests, close p < 0.001, open p > 0.5). In contrast, for the 20 Hz and no feedback conditions, the time spent closing and the time spent opening significantly deviated from both the target signal and the 90 Hz illusory kinesthetic vibration (Bonferroni-corrected chi-squared goodness-of-fit tests, χ²[2, n = 2004] = 265.2 [20 Hz v. target] 420.7 [0 Hz v. target] 107.4 [20 v. 90 Hz] 304.6 [0 v. 90 Hz], p < 0.001; Bonferroni-corrected z-tests, p < 0.011). In both of these conditions the virtual hand closed less and opened more than the target, which can be seen as a divergence of the purple and orange lines from the black target signal line (Fig. 3E). These results provide evidence that the vibration-induced kinesthetic perceptual feedback augmented the functional precision of the participant’s continuous motor output without vision. Furthermore, these results also show that the illusory kinesthetic feedback operates effectively within a myoelectric paired agonist-antagonist prosthetic control system (Movie S2).

Combining intent, illusory kinesthetic feedback, and vision establishes agency over movements

Movements are executed with an intrinsic understanding of the expected consequences of those actions(47). A sense of agency over our movements is established when we engage in a goal-directed action and sensory feedback tells us that we have completed that action(48). We used explicit and implicit measures of agency to examine the participants’ responses to combinations of intent, visual feedback, and illusory movement sensation. Questionnaires are an established method for exploring the explicit experience of agency and provide insight into self-attributions of movement(49, 50). A match between intent and sensory feedback creates a sense of agency, which can be implicitly measured as a perceptual compression of time between intent and outcome (intentional binding)(51). We displayed the virtual prosthetic hand on a horizontal monitor positioned in front of Par 1, Par 2 and Par 5 (Fig. S7A). Closing their virtual prosthetic hand to touch a virtual ball positioned at their fingertips triggered vibration through the kinesthetic tactor to induce their cylinder grip percept. Contact with the ball sounded a tone with a random delay of 300, 500, or 700 ms. The participants were told that the touch-tone time delay varied randomly over one second, and they were instructed to report their estimation of the delay on a scale from 1 to 1000 ms(50) (Fig. S7B). Seven testing conditions (Fig. S7C) were presented. At the end of each tested condition, the participants filled out a 16 question agency/embodiment questionnaire (Fig. S7D) (49, 52).

The questionnaire results showed that the vibration-induced kinesthetic perceptual feedback provided significantly greater experience of authorship over movements (agency) (linear mixed model, interaction effect of condition by question type p < 0.001, Bonferroni-corrected post-hoc t-tests each p ≤ 0.040) for a virtual prosthesis when intent, movement perception, and visual information were congruent (90Hz vibration applied, no vibration applied, and moved too fast), compared with visualizations that did not match intention or perception (delayed, moved oppositely, or moved passively) (Fig. 4A). Although voluntary contraction of the reinnervated muscle without vibration (the intrinsic motor percept) scored highly on the experience of agency, illusion-inducing vibration was required to provide effective motor control (see: 0Hz (no vibration) in Fig. 3A,B,E). A virtual prosthesis that moved faster (too fast) than the vibration-induced illusory kinesthetic percept scored significantly higher than one with an onset delay, or one that moved opposite the perceived motion, or one that was moved passively by the experimenter (Bonferroni-corrected post-hoc t-tests each p ≤ 0.037), suggesting greater attribution of agency to a virtual hand that moved more like the speed of a commercial prosthetic. Conversely, passive experimenter-driven activation of the vibration-induced illusory movement strongly reduced the experience of agency (Bonferroni-corrected post-hoc t-tests each p ≤ 0.002 for all conditions except the opposite movement, p = 0.388, which also reduced agency itself). Providing illusory movement perception did not induce a sense of limb ownership (embodiment) under any condition (score range 0.1 to −1.9); however, disagreement with statements of embodiment were significantly more pronounced when the visualized movement was opposite (opposite movement) from the vibration-induced illusory movement (Bonferroni-corrected post-hoc t-tests comparing with other conditions each p ≤ 0.048). These results provide evidence that the 90 Hz illusory kinesthetic feedback is sufficient to drive the top-down explicit experience of agency for a volitionally controlled prosthesis.

Fig. 4. Measures of agency and embodiment for combinations of intent, visual feedback, and illusory movement sensation.

(A) Average agency and embodiment questionnaire (Fig. S7D) responses across Par 1, Par 2, and Par 5 under different conditions (Baseline = illusory percept matches the hand visualization, No vibration = hand visualization closes without illusory percept, Too fast = hand visualization closes faster than the illusory percept, Too slow = hand visualization closes slower than the illusory percept, Onset delay = hand visualization closes 1 s later than the illusory percept, Opposite movement = illusory percept closes while the hand visualization opens, Passive = experimenter controlled the hand closing visualization and illusory percept; Fig. S7A,C). Error bars represent standard deviation. † indicates a significant main effect (p < 0.001) for question type (agency/embodiment vs. control) from full factorial linear mixed models (fixed effects: condition, question type). * indicates significant Bonferronicorrected post-hoc t-tests (p < 0.05) between pairs of conditions within a question type. (B) Average agency responses (n=16) compared to average estimated intervals relative to the baseline condition (n=3 intervals, 20 trials each) both averaged by condition across Par 1, Par 2, and Par 5. The horizontal dotted line denotes no difference in estimated interval from the baseline condition and the vertical dashed line indicates the +1 cutoff for an experience of agency (see: Fig. 4A). Error bars represent estimates of average standard deviation calculated as the square root of the average variance within a condition averaged across participants.

A sense of agency over movement binds together intent and outcome in time (intentional binding). Interval estimation provides a measure of agency through intentional binding where a stronger sense of agency corresponds to a shorter reported time interval. Although the intentional binding interval estimation responses for the amputee participants were not significantly different between the conditions (linear mixed model main effect for condition p = 0.179), they did share a weak negative relationship (76% confidence) with the agency responses from the questionnaires; as agency increased, interval estimations decreased (Spearman correlation, ρ(16) = −0.29, p = 0.240; Fig. 4B). Similarly to the questionnaire responses, we observed that the shortest reported delays tended to be associated with visualized movements that matched intent (baseline, too fast and too slow), and again, a hand that moved faster than the vibration-induced illusory percept produced the shortest interval estimates. The interval estimates provide support for the idea that the 90 Hz illusory kinesthetic feedback helps provide a bottom-up implicit sense of agency.

The vibration-induced kinesthetic perceptual feedback operates within clinical constraints

The standard of care for prosthetic limb attachment is a rigid socket that surrounds the residuum to mechanically couple the device to the underlying bone through the soft tissue. To verify that our approach could be translated effectively to the clinic, we implemented the perceptual kinesthetic feedback within the tight physical constraints of a prosthetic limb. We fit Par 5 with a thermoplastic prosthetic socket that incorporated EMG for control and vibratory stimulation for kinesthetic perceptual feedback. We used a dexterous robotic hand with individually powered digits to assume the identified grip conformations (Fig. S8A). When the participant signaled (EMG) hand-close, it triggered the 90 Hz illusory kinesthetic feedback for cylinder grip with the hand programmed to simultaneously move into the same conformation at a rate similar to the reported speed of the participant’s illusory percept (Fig. 5A). The robotic hand took 5.00 s to progress from fully open to the close point, where the index finger and thumb touched. For each session, the participant reported the sensation by using their intact hand to match the illusory movement percept of the missing hand(30, 31). The goal of the experiment was to see if the participant could reliably duplicate the movement of the robotic hand without being able to see it. The participant closed the robotic prosthetic hand for an average of 4.86 ±1.17 s (Fig. 5B, Y-Axis [blue]). That ending point was within 140 ms of the 5.00 s actual ideal index-thumb contact time for the robotic fingers (Fig. 5B, black crosshairs), falling within the 95% confidence interval (CI=4.46-5.27 s) which did not represent a significant difference in timing. Similarly, the participant’s other hand that was being used to match the perceived illusion started moving with the onset of their EMG control signal (−0.04 ±0.46 s) and stopped an average of 4.76 ±1.36 s later (Fig. 5B, X-Axis [red]). This ending point was within 240 ms of the actual ideal 5.00 s index-thumb contact time of the prosthetic fingers (Movie S3). These results demonstrate that the 90 Hz kinesthetic perceptual feedback can be used to reliably duplicate the movement of an unseen prosthetic hand in the context of a clinical prosthetic limb.

Fig. 5. Application of kinesthetic illusory feedback within a bidirectional neural-machine-interface.

(A) Schematic representation of the movement feedback paired to a real-time functional prosthetic hand clinically fitted to the participant with illusory feedback locked to their volitional control, which was used to explore clinical feasibility. Feedback pathways are represented in blue (VCLM = Voice Coil Linear Motor). Prosthesis control pathways are represented in red (participant control). Participants matched the perceived sensation with their intact hand, and prosthetic hand closing speed was timed to the demonstrated perceptual illusion. (B) Graph showing the average start/stop times of the control signal (n = 32, EMG-activated prosthetic hand closing, blue) and the average start/stop times of the concurrently demonstrated percept movement (n = 32, matching hand, red) superimposed over the ideal start and thumb-index finger contact times of the physical prosthetic hand under continuous drive (black crosshairs, radius = 250 ms). The 5 s progression of movement from fully open to thumb-index finger contact for the physical prosthetic hand is our approximation of the participant’s demonstrated movement of the illusory percept (grey dashed lines). All events are plotted along time-linked axes. The raw matching hand movement start times (solid circles) and stop times (x’s) are colored according to relative position within the experimental timeline (first = blue, last = red). Gold stars = intersection of the average movement start and stop points for the EMG-control and the demonstrated movement. (C) Graph showing average percept speeds (n=30; error bars represent standard deviation) in Par 1, Par 2 and Par 5 measured before (Pre), after one, two, three, four and five conditioning games (ASG1-5) designed to increase percept speed (Fig. S8B,C), and after a washout period (Post). The grey area represents the range of hand close speed at the lowest speed setting in a common commercially available prosthetic hand (OttoBock Speed 0 Range).

With all of the experimental approaches we found that Par 1, Par 2, and Par 5 were able to contract their myosites to control the virtual and physical prosthetic hands while simultaneously receiving illusory feedback. However, muscle vibration has been shown to reflexively activate muscle contraction which could confound clinical implementation(53). We analyzed EMG recording data from the first set of Passive (90 Hz vibration) and Active (volitional muscle contraction with 90 Hz vibration) trials, compared to baseline EMG (the quiescent period before application of vibration/contraction) for Par 1, Par 2, and Par 5 during the intentional binding experiments (Fig. S9). With the application of 90 Hz vibration there was a slight increase (non-significant for Par 1 and 2; Bonferroni-corrected t-tests, Par 1 p = 0.072, Par 2 p > 0.5, Par 5 p < 0.001) in average muscle activity over baseline (1.4× above baseline, Fig. S9A). However, this effect was significantly lower (Bonferroni-corrected t-tests each p < 0.001) than the average level of normal volitional contraction used to drive the EMG control signal (7.5× above baseline, Fig. S9A). Although reflexive contraction may be responsible for increased muscle tonal activity during 90 Hz vibration this signal is negligible in comparison to a normal muscle contraction and does not impact volitional EMG control (Fig. S9B).

The passively applied kinesthetic illusion moved slowly (7 mm/s). Although the actively triggered kinesthetic percepts were an average of 85% faster (52 mm/s, Bonferroni-corrected t-tests each p < 0.001) (Movie S1), they were still at the low end for the closing speed of a typical prosthetic hand (thumb to index closing speed of 15-100 mm/s at speed setting 0; OttoBock MyoHand VariPlus Speed). Visualizations of different hand movement speeds have been shown to influence the perception of the kinesthetic illusion(54). To determine if we could increase the movement speed of the perceived kinesthetic illusion, we developed a paradigm to condition Par 1, Par 2, and Par 5 to visually couple their perceived illusion to faster operation of the virtual hand (Fig. S8B). The participants were instructed to close their virtual hand onto a small ball hovering between the fingertips as fast as they could before it turned red (Movie S4). The ball started out black and then turned white to signal the participants to start closing the virtual hand. Importantly, they did not actually control the speed of their hand. Instead, successfully catching the ball to turn it green was based on a reaction time grace period (≤ 500 ms) plus the time allowed for the speed of the hand close visualization itself. Failure on the trial resulted from either starting too soon or too late and the reaction time grace period was adjusted so that they were not always successful but did not fail so often that they lost interest (Fig. S8C) (Movie S4). The participants were unaware that 1) they did not have control over their virtual hand speed, 2) success or failure was tied only to reaction time, and 3) that every 30 trials the virtual hand visualization increased in speed, and when the speed could not be increased any further the grace period was reduced (Fig. S8C). After each set of trials, the participants matched their actively triggered illusory percepts with their intact hand while wearing the data glove (Movie S4). The data glove trajectories were used to calculate the demonstrated percept speeds and compare them to a pre-conditioning baseline and the speed of a commercial prosthetic hand (OttoBock MyoHand VariPlus Speed). This conditioning game increased the active peak percept speed in all three of the amputee participants (Bonferroni-corrected t-tests first vs. fastest percept captured, Par 1 p = 0.037, Par 2 and Par 5 p < 0.001) (Fig. 5C). Par 5 showed the most dramatic change, with a peak post-conditioning speed that exceeded our capacity to make the virtual hand visualization close any faster. Par 1 showed a small initial increase in peak speed but rapidly fatigued and was unable to continue the game. However, for the two participants who were able to complete the task (Par 2 and Par 5) the peak speeds were commensurate with speeds 0 and 1, respectively, of the commercially available prosthetic hand and Par 5’s percept speed remained high after a washout period (Bonferroni-corrected t-tests first vs. washout percept captured, Par 2 p = 0.171 and Par 5 p < 0.001). These results demonstrate that the vibration-induced kinesthetic percepts can be rapidly visually updated to reach speed scales that are similar to commercially available prosthetic hands (Movie S4).

Discussion

Successful use of advanced prosthetic limbs requires both effective motor control and sensory feedback. This study closes the prosthetic feedback loop by harnessing the kinesthetic illusion to provide relevant higher-order input to human amputees about the ongoing movements of their artificial hands.

All the targeted reinnervation amputees tested reported kinesthetic percepts. They also independently reported similar complex synergistic movement percepts that appear to reflect basic coordinated actions (movement primitives, synergies or engrams) required for dexterous manipulation(55). In our work and in sensory-neural stimulation studies for prosthetic feedback we found that similar stereotypic grip conformation percepts were often reported such as fist closing (cylinder grip), tripod grip, and thumb-index fine pinch(15, 56, 57). In peripheral sensory-neural stimulation studies, subsets of afferents within nerves are stimulated through electrodes. Similarly, in targeted reinnervation there is competition between the regenerating neurons for a limited number of neural targets in the denervated muscle and skin(58, 59). In these neural feedback approaches there appears to be limited muscle sensation information returning to the brain through the interface as compared to able-bodied nerves; yet, the amputee participants across these studies describe stereotypic complex multi-digit synergistic hand conformations. This suggests that the brain may be attempting to contextualize the reduced sensory inputs in terms of basic modules of coordinated activity. Motor control and movement sensation face similar organizational challenges. Limbs and hands are capable of infinite combinations of movements to achieve any singular goal(60). The brain appears to simplify this infinitely high dimensionality by organizing movement control based on assemblages of basic coordinated output patterns(61). The results we present here provide evidence that the brain representational architecture of kinesthesia likely shares similar organizational properties with movement production.

Although evidence suggests that cutaneous innervation plays a key role in kinesthesia(37, 62), we found that the kinesthetic illusory percepts were generated in participants in this study without cutaneous tactile sensation. Furthermore, among the participants in this study with both motor and sensory reinnervation, there was no consistent relationship between cutaneous touch in the skin and movement percepts in the deep muscle. In able-bodied individuals, the illusory perception of movement from muscle vibration is most often attributed to signals arising from muscle elongation(33). However, in the targeted reinnervation amputees, vibratory stimulation of muscles reinnervated by the median nerve produced illusions of flexion (not extension) whereas muscles innervated by the radial nerve produced illusions of extension (not flexion). This result has also been demonstrated in neural stimulation studies in amputees where stimulation of the median nerve produced percepts of digit flexion(63, 64). In both situations, the sensory stimulation giving rise to the movement percepts supports the idea that muscle afferents signaling active contraction may play a key role in kinesthesia.

Control of hand grip aperture is an elemental movement requirement for skilled prosthetic function. The participants showed high resolution of grip aperture using the vibration-induced kinesthetic illusory feedback. The ability of amputee participants to couple illusory kinesthetic feedback with active control bodes well for restoring more normal reach-and-grasp functions. We also demonstrated the feasibility of having 2 tactors effectively running in agonist-antagonist muscle pairs during bidirectional grip aperture tracking, which confirmed the ability to employ more than a single percept during continuous movement control; although this was only within the limitation of a single degree of freedom (grip open/close). Ideally, future work will explore incorporating additional joint movement percepts to represent multiple degrees of freedom. Importantly, we also showed that adding vision to the vibration-induced kinesthetic perception would likely enhance the control strategy. Kinesthesia provided better temporal resolution than vision, and the best resolution was provided by vision and kinesthesia together. The illusory kinesthetic feedback appears to work optimally when coupled with vision, as would occur in an intact person, which provides evidence that the kinesthetic perception provided by muscle vibration can functionally integrate with existing natural sensory feedback.

Beyond motor control, the combination of vision and perceptual feedback has implications for the critical cognitive elements that determine the extent to which humans will integrate with machines. Proprioception, touch, and vision integrate with each other to establish the awareness of self versus other. Agency (the experience of authorship of one’s movements) appears to arise from the integration of the internal model, vision, and movement sensation(19). By contrast, embodiment (the experience of owning one’s body) arises primarily from the integration of vision and touch(65). Agency and embodiment both appear to contribute to a complete sense of body awareness(66); however, studies investigating whether or not movement and agency can help to enhance embodiment show contrasting results(49, 66). Investigations of the visual motor contribution to the experience of embodiment are constrained by using able-bodied individuals, invariably requiring tracking the movements of hands and digits that have intact cutaneous tactile sensation(66, 67). On the contrary, the neural-machine interfaces employed in this study allowed access to kinesthesia without touch sensation, therefore providing a unique opportunity to investigate the contribution of movement perception itself. Within this model system, we found that the perception of kinesthesia alone elicited an experience of agency but did not provide a sense of embodiment.

Agency has special relevance for the clinical implementation of advanced prosthetic limbs because these devices are computerized machines, with which the amputee must cooperate to complete tasks. In human-human cooperation, the subjects are able to form a ‘unified agency’ over movements(68). In human-computer partnerships, however, evidence suggests that the neural processes responsible for establishing agency are inhibited(69). Current prosthetic limbs can incorporate capabilities that do not include the user in the control loop, such as auto-grip slip detection. However, autonomous device functions that run outside of the user’s control are found to be frustrating to interact with(70). As advanced prosthetic limbs progress and become even more autonomous in their operation, establishing an intrinsic sense of agency for these devices through perceptual integration will be key to user acceptance and realization of the full functional potential of computerized prosthetic limbs within a cooperative joint-agent partnership.

We report here that differences in the speed of the visualized hand movements influence the experience of agency. This is compelling because there was concern that a slow percept would prevent appropriate utilization of the vibration-induced movement percepts for prosthetic hand feedback. However, it appears that the opposite may be true. The visual update reflecting a faster hand not only effectively induced a sense of agency, but when coupled with the results of the speed game, it increased the speed of the vibration-induced movement percept. These results have direct implications for the clinical implementation of perceptual movement feedback; during regular prosthesis use, the prosthetic hand would not need to be slowed to match the speed of the illusion. Instead, a prosthetic hand that moves quickly will likely be readily incorporated into the representational structure of the body schema, thus linking perception effectively to a hand speed that is typical of commercially available prostheses.

The illusory kinesthetic percepts were found in all six recruited participants in the initial mapping experiments. However, due to constraints from increasing demands on participant time, with each subsequent round of experimentation, we were unable to complete testing in the full amputee cohort. Statistical evaluation was limited by the small number of participants (n=3) who could commit to the time and travel required to complete the full study protocol. The complexity of implementing simultaneous feedback and control in a prosthetic system also presented technical challenges to study design. We focused on hand function first because the illusory percepts were primarily hand focused and hand grasp is the most functionally relevant feature of upper limb prosthetic use. However, it has yet to be shown that kinesthetic feedback will improve prosthetic performance in daily activities. This will require extending beyond the virtual reality approaches that were used here. Furthermore, the inclusion of additional feedback from multiple limb joints such as the wrist and elbow will be crucial to the functional use of wearable advanced devices and additional clinical testing will be essential to assess impact. This will require further refinement and miniaturization of kinesthetic feedback tactors to provide stimulation at multiple sites for induction of a greater number of movement percepts. It will also require refining and incorporating new methods of simultaneous multi-joint motor control(71). Furthermore, this study also did not include touch feedback which will be necessary for full utilization of fine finger control, object manipulation, and functional stereognosis. However, as with the simultaneous implementation of kinesthesia and motor control, the addition of touch feedback further increases system complexity. Currently we have no evidence to suggest that the vibration-induced illusory kinesthetic feedback is incompatible with tactile sensibility resulting from targeted reinnervation or any combination of kinesthesia, touch, or motor control in this neural-machine interface system. Current work is continuing to investigate multimodality integration and testing for motor control, kinesthesia, and touch. Further studies of movement percepts for other levels and locations of amputation and in amputees without a neural-machine interface are also ongoing.

In this study, we provide evidence that the vibration-induced illusory percepts have direct impact on mechanisms of motor control and induce agency with improved functional motor performance. This kinesthetic feedback does not need to be learned: turning it on allows the amputee participants to function indistinguishably from able-bodied individuals in grip aperture control, and turning it off abolishes the effect. The feedback system operates within the fitting constraints and control strategies of standard-of-care prosthetic limbs, and the results show that with regular use the movement percepts will likely adapt to the movement speed of commercial prosthetic hands. The approach reported here is clinically feasible and provides a critical new element that could lead to fully-integrated and efficient bi-directional prosthesis control. In the near future, joining together kinesthetic, cutaneous, and motor systems could result in cohesively integrated fully bi-directional prosthetic limbs that are intuitively controlled, cognitively embodied by touch(52), and provide a natural perceptual sense of complex artificial hand movement.

Materials and Methods

Study Design

Our hypothesis was that we could elicit kinesthetic illusions of movement in the missing limb of amputee participants who had undergone targeted reinnervation, to provide relevant closed loop feedback for prosthetic control. The goals were to (i) discover whether movement illusions could be elicited in the missing limb; (ii) investigate functional use of the illusions and the impact on agency and internal dynamic model; and (iii) demonstrate clinical implementation. For the percept mapping, we recruited 6 upper limb amputee participants that had undergone targeted reinnervation(24–26). (Fig. S1). For the psychophysical quantification, there was drop out of one participant due to time constraints. For the second set of experiments, a subset of 3 participants were available for the required days of testing (of the original 6 participants, one could not participate due to health issues, and the other 2 had travel and time constraints limiting availability). During these visits, we investigated the illusory hand kinematics; grip aperture matching; adaptation and just noticeable difference; intentional binding; speed game and agency experiments. For the adaptation experiments, data from one participant (Par5) was not able to be analyzed due to a change in experimental design, so results from 2 participants are provided. The last experimental goal investigating clinical implementation with bidirectional feedback required fitting a physical prosthetic device, and only one amputee was able to participate due to travel and availability constraints for the others. This participant also underwent the additional agonist-antagonist hand tracking experiment (Fig. S6).

All experiments were conducted according to Declaration of Helsinki principles under the approval of the local institutional research ethics boards (IRB or ethics) at Cleveland Clinic, the Louis Stokes Cleveland VA Medical Center, the University of Alberta, and the University of New Brunswick. All amputee participants had previously undergone targeted reinnervation(24–26)(Fig. S1).

Statistical analysis

Two-tailed t-tests for each participant compared means of measures dependent on one factor with two or three levels (Bonferonni-corrected for multiple participants and levels). Measures dependent on more than one factor or factors with multiple levels were first pooled across participants and fit with a linear mixed model with fixed effects for each factor and interaction, a random participant-specific intercept, and equal or unequal covariance between trials and trial variance depending on the structure that best fit the data as determined by Akaike’s information criterion. Bonferroni-corrected post-hoc t-tests explored significant main and interaction effects. Pearson correlations quantified similarity between trajectories and were averaged across conditions to create an overall measure of similarity. Root-mean-square differences quantified alignment between pairs of trajectory vectors or response values and were averaged across pairs for an overall measure of alignment. Chi-square goodness-of-fit tests and follow-up z-tests between pairs quantified differences in proportions with Bonferonni correction across factor levels for the total number of chi-square tests plus follow-up z-tests. Comparisons were considered significant when p < 0.05. For details on statistical analyses for each experiment, please see Supplementary Materials and Methods.

Supplementary Material

Movie 01

Fig. S1. Overall participant demographic, surgical, and experimental details with comparisons of cutaneous touch percepts in the skin with underlying movement percepts in the muscle.

Fig. S2. Perceived magnitude of the kinesthetic illusion.

Fig. S3. Virtual or prosthetic hand movement and linked vibration-induced kinesthetic percept, controlled either actively (by the amputee) or passively (by the experimenter).

Fig. S4. Grip aperture experimental setup and percent close grip values for able-bodied cohort on a single degree of freedom task.

Fig. S5. Falling block experimental setup and example analyses used in adaptation and just noticeable difference experiments.

Fig. S6. Clinical implementation with two-site (agonist-antagonist) kinesthetic feedback.

Fig. S7. Setup for intentional binding experiments, conditions tested, and questionnaire statements.

Fig. S8. Matching complex hand percepts to the dexterous robotic prosthetic hand.

Fig. S9. Muscle activity (EMG) relative to baseline when virtual hand movement and linked vibration-induced kinesthetic percept were controlled passively (by the experimenter) or actively (by the amputee).

Movie S1. Demonstration of active and passive percepts.

Movie 02

Movie S2. Clinical implementation of two-site agonist-antagonist kinesthetic feedback.

Movie 03

Movie S3. Volitionally-controlled, clinically-fit robotic hand matching perceived movement.

Movie 04

Movie S4. Speed game demonstration.

One Sentence Summary.

A perceptual illusion provides the sensation of complex bionic hand movements to human amputees, allowing real-time movement control without the necessity of vision.