Dexterous Control of a Prosthetic Hand Using Fine-Wire Intramuscular Electrodes in Targeted Extrinsic Muscles

Abstract

Restoring dexterous motor function equivalent to that of the human hand after amputation is one of the major goals in rehabilitation engineering. To achieve this requires the implementation of a effortless human–machine interface that bridges the artificial hand to the sources of volition. Attempts to tap into the neural signals and to use them as control inputs for neuroprostheses range in invasiveness and hierarchical location in the neuromuscular system. Nevertheless today, the primary clinically viable control technique is the electromyogram measured peripherally by surface electrodes. This approach is neither physiologically appropriate nor dexterous because arbitrary finger movements or hand postures cannot be obtained. Here we demonstrate the feasibility of achieving real-time, continuous and simultaneous control of a multi-digit prosthesis directly from forearm muscles signals using intramuscular electrodes on healthy subjects. Subjects contracted physiologically appropriate muscles to control four degrees of freedom of the fingers of a physical robotic hand independently. Subjects described the control as intuitive and showed the ability to drive the hand into 12 postures without explicit training. This is the first study in which peripheral neural correlates were processed in real-time and used to control multiple digits of a physical hand simultaneously in an intuitive and direct way.

I. Introduction

The restoration, following amputation, of dexterous control equivalent to that of the human hand is one of the major goals in applied neuroscience and bioengineering [1]–[5]. To accomplish this requires achieving two important subgoals: the development of a multi-degree of freedom (DoF) artificial hand [6], [7] and the implementation of an intuitive and effortless human–machine interface (HMI) that maps the sources of volition to the DoFs of the artificial hand. The HMI is of interest for this work.

Many attempts to tap into the neural signals underlying voluntary control have been made, ranging in invasiveness (from neural implants to surface sensors) and hierarchical location (CNS, PNS, skeletal muscles). Although significant research efforts have been made with other approaches (see Micera et al. for review [8]) the most reliable and clinically viable technique today remains the use of the electromyogram [(EMG), i.e., the electrical activity produced by skeletal muscles as a by-product of normal muscle contraction], picked up by surface electrodes to control the movements of an electromechanical prosthesis. This type of control can be slow and unintuitive especially in the case of multi-DoF prosthetic arms and hands. For multi-DoF arms the different DoFs are controlled in a sequential fashion, using locking mechanisms and/or special switch signals to change control from one DoF to the next. Prosthetic hands are typically controlled by a single agonist/antagonist EMG pair that controls hand opening and closing and thus the fingers cannot be controlled individually.

Depending on the level of amputation, individuals with a below-elbow (i.e., transradial) amputation maintain a number of the 18 extrinsic muscles that originally articulated the fingers and wrist. Hence, EMGs from selected extrinsic muscles could be used to control physiologically appropriate DoFs in a multi-digit hand prosthesis. EMGs from specific muscles can be collected using intramuscular techniques, such as chronically implanted myoelectric sensors (IMESs) [5] or by needle or fine-wire electrodes inserted into target muscles [9] for acute recordings. The pickup volume of these electrodes is localized to the targeted muscle and the recorded signal is free from crosstalk from neighboring muscles [10]. Thus each electrode signal could be used as a direct and independent control input for one DoF (e.g., one finger) of a prosthesis. We hypothesized that multiple muscles targeted by intramuscular electrodes could be used to achieve independent and simultaneous control of multiple fingers of a myoelectric prosthesis. However this is a complicated, and still not fully understood, problem. In the neuromuscular system when individual finger movements are attempted there is unintended activity of other muscles that can cause movements of the other fingers and/or the wrist to occur [11]–[15]. In addition to neural interactions, there are mechanical connections and interactions between muscles that cause unwanted motions. Therefore, it is still unknown whether multiple intramuscular EMGs from extrinsic muscles could be used to directly control a multi-fingered prosthesis. The recent work by Birdwell et al. [16], [17] suggested that this is possible. They implemented a three digit control paradigm (i.e., flexion/extension of thumb, index and middle) and tested both proportional speed control and a pattern recognition controller in a virtual posture-matching task, using intramuscular recordings.

In the present work we assessed for the first time whether intramuscular recordings from the extrinsic flexor muscles could be used to simultaneously and independently control, in real-time, four DoFs of a physical robotic hand (i.e., abduction/adduction of the thumb and flexion/extension of thumb, index and middle fingers). Based on the work by Birdwell et al. [16], [17], the Flexor Pollicis Longus (FPL), FDP₁, FDP₂ (i.e., the first two compartments of the Flexor Digitorum Profundis) and Abductor Pollicis Longus (APL) were targeted with intramuscular electrodes and their EMG proportionally mapped to the flexion of thumb, index, and middle digits and to the abduction of the thumb in the multi-DoF prosthetic hand, respectively. We hypothesized that if the correct muscles were targeted then physiologically appropriate, simultaneous multi-DoF control could be achieved, without explicit training. Four healthy individuals participated in the experiments. The first experiment was aimed to assess simultaneous multi-DoF controllability and subjects were asked to perform 12 hand postures by directly controlling the prosthetic hand. The second experiment was a sinusoid-tracking task and was aimed at addressing the viability of the present HMI under dynamic conditions. Unfortunately, actual grasps with the robot hand could not be assessed as the percutaneous wires impeded our subjects from comfortably performing reaching movements.

Our results demonstrate the feasibility of achieving simultaneous, independent and continuous control of individual digits on a prostheses directly using extrinsic muscle EMG signals. These results open up promising possibilities for individuals with transradial amputations since there are a number of multi-digit prosthetic hands commercially available (like the Bebionic v2 by RSL Steeper or the iLIMB by Touch Bionics) and chronically implanted IMES [5] are becoming the reality [18], [19].

II. Materials and Methods

A. Participants

Four right-handed able-bodied subjects participated in this study. They were free of any neurological or motor disorders. Two males aged 23 and 27 volunteered in preliminary experiments (experiment 2B, subject S1 and S2); two other males aged 29 and 35 participated in the core experiments (experiment 1 and 2A, subjects S3 and S4). The study was approved by the University of Colorado at Boulder Institutional Review Board and informed consent was obtained from each subject.

B. Fine-Wire Electrodes and Target Muscles

Fine-wire intramuscular EMG recordings [9], [20] permit the local measurement of EMG activity from within target muscles for a limited time without the need for a surgical procedure. Bipolar percutaneous electrodes are inserted into the belly of the muscle of interest using Basmajian’s single needle technique [20]. This technique enables isolated measurements from individual muscles despite the depth of the muscle and surrounding musculature. The detection volume of a bipolar electrode approximates a sphere with diameter equal to the spacing between the leads [21]. Our electrodes (~ 0.05 mm diameter) had exposed leads placed a few millimeters apart (<2 mm bipolar electrode spacing); thus the pickup volume was smaller than 5 mm³.

In the present work, four muscles were selected as the control sites: the first and second compartments of the FDP (i.e., FDP₁ and FDP₂) for the fingers, and the FPL and APL for the thumb. These muscles were chosen so they could control corresponding DoFs in our robotic prosthesis; the DoFs were selected because they could drive the hand to form hand postures useful in daily living [6], [7], [22]. FDP is a compartmental muscle that is comprised of four tendons that insert in each of the fingers. Although it is a single muscle, its four compartments (FDP₁ … FDP₄) can be activated relatively autonomously to flex each of the four fingertips [13]. FPL primarily causes flexion of the thumb’s inter-phalangeal joint and is situated in the same deep layer as FDP and inserts in the distal phalanx of the thumb. The APL lies deep in the posterior compartment of the forearm inserting into the radial side of the base of the first metacarpal bone and causes thumb abduction at the carpometacarpal joint. All four muscles lie in the proximal third of the forearm and can potentially be targeted after a transradial amputation.

C. Myoelectric Controller

Raw fine-wire EMGs were collected using a Noraxon TeleMyo 2400R (Noraxon, Scottsdale, AZ, USA) through a wireless unit (TeleMyo 2400T). A reference electrode was placed on the lateral epicondyle. Raw data were hardware filtered (high pass: 10 Hz–1st order; low pass: 1000 Hz–8th order; gain: 5000), acquired via a data acquisition board (PCI-6221 National Instruments) on a PC (sampling frequency: 3 kHz; resolution: 12 bits) and processed in real time by a custom LabView software application (National Instruments, Austin, TX, USA). Signals were filtered using a 6th-order Butterworth band-pass filter (10–500 Hz) and a notch filter (60 Hz). The mean absolute value (MAV) was then computed by a moving average filter (nonoverlapping windows) which produced a new sample every 100 ms. The MAV from each channel was linearly mapped to a set-point for the corresponding DoF of the prosthetic hand, such that ~70% maximum voluntary contraction (MVC) corresponded to full flexion (or abduction) of the DoF and no contraction corresponded to full extension (or adduction). Hence every 100 ms the software application sent a new hand-posture Pt to the robotic hand over a serial bus [Fig. 1(a)]. The output rate of 10 Hz and the time response of the controller (measured <100 ms) were fast enough that the subjects could not perceive the delay, in agreement with the literature [23]. In addition to controlling the hand during the experiments, the software also presented visual guidance and cues to the subjects using a computer screen as well as recorded all data for offline analysis.

Fig. 1.

Experimental setup. (a) Block diagram of the myoelectric controller. FPL: flexor pollicis longus. FDP₁, FDP₂: first and second compartment of the flexor digitorum profundis. APL: abductor pollicis longus. ADC: analog to digital converter. MAV: mean absolute value. Pt: posture control command sent to the robotic hand. (b) Subject wearing an orthopedic splint on the experimental hand sat in front of a computer screen and the robotic hand. Computer screen presented desired posture cues while the hand was controlled in real-time. (c) Pictures of the 12 target postures P1..P12 used in experiment 1. Last two fingers of the hand are not shown for clarity (since these were not under direct control). It is worth noting that the target position of the fingers in postures P5 and P10 are similar but the two postures required activations with different timing in order to position the thumb properly (under the index and middle in posture P5; over the index and middle in P10). (d), (e) Representation of the sinusoids that subjects tracked in experiment 2A–B, respectively.

D. Artificial Hand

The robotic hand was a right-handed, commercial version of the SmartHand [6] (Prensilia S.r.l., Pisa, Italy, cf. Fig. 1). It consisted of four fingers and a thumb actuated by five electrical motors. Allowed motions were flexion/extension of the thumb, index, and middle and rotation of the thumb opposition space (hereafter abduction/adduction) and were controlled by EMG activity from FPL, FDP₁, FDP₂ and APL, respectively. The hand includes encoders on each motor and an electronic controller that implemented position control, by receiving commands sent over a serial bus from the PC. The hand is able to close in less than 2.0 s at full speed and its time response is negligible with respect to the time response of the myoelectric controller used in this work.

E. Experimental Protocol

Subjects were instructed to make appropriate test contractions during electrode insertion and the EMG signal was played through a speaker to help locate the desired muscle or compartment. The hypodermic needle was removed after the muscle was located, leaving the fine-wires residing in the target muscle belly. A constant current stimulator (Digitimer Ltd. Model DS7A, Hertfordshire, U.K.) was then attached to the fine-wire electrodes and was used to elicit a short muscle contraction. The resultant motion of the digit was used to confirm the proper electrode placement. The wires were then shortened so only a small length of wire remained to connect to the wireless data acquisition system. These were taped to the skin to minimize motion artifacts.

After the electrodes were correctly inserted, the subject donned an orthopedic hand splint to enable him to generate isometric muscle contractions (strong EMG activity can be more comfortably produced when contracting against a physical restraint) and to standardize the test posture across subjects. The MVC was recorded from each muscle/compartment (while the other muscles were kept relaxed). Each channel was thus independently adjusted so that ~70% MVC mapped to full flexion (100% flexion) of the corresponding DoF in the prosthesis and relax activity mapped to no-flexion (0% flexion). Two experiments were then performed with the subject seated in front of the robotic hand and the computer screen [cf., Fig. 1(b)].

Experiment 1

Subjects were asked to form and hold 12 hand postures that involved the four DoFs selected [cf., Fig. 1(c)]. The 12 postures tested the full range of motion (RoM) of the selected four DoFs in the prosthetic hand and more importantly, resembled functional postures and grasps used in activities of daily living [22]. The hand was controlled in real-time during this experiment using the position control scheme described previously. Pictures showing the target posture were presented to the subject on the computer screen. After each picture was presented, a light on the screen turned green to prompt the subject to form the posture. The delay between picture presentation and “green-light” was randomized between two and four seconds; this unknown delay insured that subjects started to form the posture only when prompted. Subjects were given 10 seconds from the time of the “green-light” cue to form and hold for 0.5 seconds the prompted posture. Postures were considered to be successfully formed if all of the four DoFs were simultaneously positioned within a ±15% envelope of the target posture Pt (where 100% corresponded to the whole RoM) and held for 0.5 seconds. If a posture was successfully formed or if the 10 seconds elapsed the next picture/posture was presented. To aid with target posture acquisition visual feedback in the form of yellow and green lights (on the screen) turned “on” if the corresponding DoF was within the ±10% or ±5% of the target posture envelope, respectively. The 12 postures were repeated in five consecutive blocks. Subjects were given a five-minute break between consecutive blocks to rest and limit fatigue.

Four metrics were used to quantify performance. The completion time (Tc) was the time taken to successfully reach and hold a posture for 0.5 seconds. This quantity represented how quickly EMG command information could produce accurate hand postures. The completion rate (CR) was defined as the percentage of successfully completed postures. This metric was a measure of real-time performance reliability. The Tc and CR were valid if the posture was successfully completed within 10 s. The minimum posture error (MPE) was the minimum difference between controlled (actual) posture and target (desired) posture, calculated over the trial as a percentage of the full span of the four DoFs. The End-of-trial mean Posture Error (EPE) was the mean posture error calculated in the last third of the trial (from t = 6.6 to t = 10 s). Both MPE and EPE were computed for the unsuccessful trials; successful trials ended early. The MPE and the EPE were of interest because they provided information on how far the unsuccessful trials were from the target.

Experiment 2A

The second experiment was a real-time tracking task aimed to assess dynamic performance of the HMI and quantify the effective independence of the four EMG channels. Subjects had EMG-modulated position control of the Y-axis coordinate of a cursor on the screen and were instructed to trace scrolling sinusoidal waveforms. Subjects tested one of the four muscles (FDP₁, FDP₂, FPL or APL) at a time while keeping the other (nontest) muscles relaxed. The cursor was controlled using the same calibration used in experiment 1 and all EMGs and outputs (i.e., the position of the DoFs) were recorded for offline analysis. Four sinusoids with different features (mean value, amplitude, period, and speed) were used, as described in Table I [Fig. 1(d)]. These sinusoids were chosen as we had found in pilot tests, that they were very challenging to track even when using a data-glove as the input. Each participant tracked the four sinusoids using each of the four muscles/compartments for a total of 16 trials. The duration of each trial was 40 seconds with a two-minute break after each trial.

TABLE I.

Features of Sinusoids Used in Tracking Experiment

Type (acronym)	Mean [% RoM]	Amplitude [%RoM]	Period [s]	Speed [%RoM/S]
Fast, half range (FA)	50	50	2 s	25
Slow, full range (SA)	50	100	6 s	16.6
Lower half span (LA)	25	50	6 s	8.3
Higher half span (HA)	75	50	6 s	8.3
Lower half span (LB)	25	40	7.7	5.2
Middle half span (MB)	50	40	7.7	5.2
Higher half span (HB)	75	40	7.7	5.2

Experiment 2B

In preliminary exploratory experiments two subjects (S1 and S2) performed the tracking task following the same procedure but with another set of sinusoids [Fig. 1(e)]. These sinusoids were slower compared to those used later in experiment 2A by subjects S3 and S4 and therefore easier to track accurately. Amplitude and frequency of the position sinusoids were fixed: the amplitude (i.e., the RoM the DoF had to cover) was 40% of total RoM; the frequency was 0.13 Hz (period: 7.7 seconds). The mean value of the sinusoids was varied among 25% (L_B), 50% (M_B), 75% (H_B) of full RoM as described in Table I. Each participant tracked the three sinusoids using each muscle or compartment (FDP₁, FDP₂, FPL or APL), for a total of 12 trials.

For both experiments 2A and B the mean absolute error (MAE) between the target position, Pt, and the controlled position of the intentionally driven DoF, Pi, was used as the performance metric; the MAE was normalized to the full span of the DoF (P_MAX)

$$\text{MAE}(\%)=\frac{{\displaystyle \sum _{t=t_0}^{t_F}}\mid {Pt}_t-{Pi}_t\mid }{(t_F-t_0)·P_{\text{MAX}}}.$$ (1)

The spurious activity from the nontest muscles or compartments that yielded to unintentional movements of the other DoFs was also accounted for by computing the relative mean activity (RMA) and the relative variance activity (RVA) for each trial [16]. The RMA of a nontest DoF was defined as the average ratio between the unintentionally controlled position of the DoF, Pu and Pi, normalized to the full span of the DoF

$$\text{RMA}(\%)=\frac{{\displaystyle \sum _{t=t_0}^{t_F}}{Pu}_t/(t_F-t_0)}{{\displaystyle \sum _{t=t_0}^{t_F}}{Pi}_t/(t_F-t_0)}·\frac{1}{P_{\text{MAX}}}.$$ (2)

The RVA was defined as the ratio between variances about the mean of Pu and Pi

$$\text{RVA}(\%)=\sum _{t=t_0}^{t_F}\frac{\left|{Pu}_t-\frac{{\displaystyle \sum _{t=t_0}^{t_F}}{Pu}_t}{(t_F-t_0)}\right|}{\left|{Pi}_t-\frac{{\displaystyle \sum _{t=t_0}^{t_F}}{Pi}_t}{(t_F-t_0)}\right|}·\frac{1}{P_{\text{MAX}}}.$$ (3)

The RMA and RVA quantified the mean independency and dynamic independency of the intentionally driven DoF with respect to the nontest DoFs, respectively. All the metrics were calculated in a time-window from ten seconds after the beginning of the trial (t₀ = 10 s) to the end of the trial (t_F = 40s). The first ten seconds were considered as a transient period: during this time the position signal (Pi) had to catch up with the target (Pt) after starting from a position equal to 0% flexion.

III. Results

A cross-correlation analysis performed on the data verified that the EMG measurements of each electrode were representative of individual muscle compartments and were not corrupted by muscle crosstalk. This analysis determined that the correlation coefficient between any pair of muscle activity signals was <0.1. Previous work has shown that values lower than 0.3 represent signals that are not contaminated with muscle cross talk [21], [24]. This was an important distinction to make because electrodes were in adjacent muscle compartments separated by only a few millimeters. Consequently, any simultaneous activities measured during the experiments were believed to be neurological coactivations of muscle/compartments or muscle simultaneous contractions and not muscle crosstalk between EMG sensors [9], [16].

To illustrate the experiments, we provided two video clips as supplementary material, showing the experimental setup while subject S3 controls the robotic hand, along with EMG signals.

Experiment 1

The graphs in Fig. 2 show the raw EMG signals for each DoF associated with acquiring a representative target posture (i.e., P3—a palmar grasp). The graphs also show the actual position taken by the controlled DoFs, the target posture with the ±15% envelope and the completion time (Tc). The hand was controlled in real-time during this experiment. It is interesting to note that different DoFs were volitionally controlled in different times even though the controller allowed for simultaneous and coordinated control of all four DoFs. For this representative trial, the thumb abduction was moved and held in its target position first and then the flexion DoFs were activated and controlled simultaneously until the end of the trial. This behavior, together with the presence of “bursts” in the EMG traces, denotes voluntary modulation of the different DoFs. The histograms in Fig. 3(a) show the individual performance metrics for subjects S3 and S4 in carrying out the experimental task. Subjects’ completion rates (CR) were different and in general subject S3 (53±11%, mean ± standard error of mean) achieved better results compared to S4(35±7%). In particular S3 was able to perform at least seven target postures with CR = 60% whereas S4 could not complete more than five postures with CR = 40%. In addition, each subject was not able to achieve specific postures: P7 and P8 for S3 and P4, P7, P11 for S4. The time metric, Tc demonstrated little variability within postures and the aggregate result was consistent between subjects (58 ± 0.8 s for S3 and 60 ± 1.6 s for S4). This result suggests that when they could be achieved postures were completed in a posture-specific time that was independent of the trials.

Fig. 2.

Posture matching experiment representative trial. Recorded EMG signals (in black—left Y axis) and actual position of the DoF in the robotic hand (in blue—right Y axis) for the four DoFs, in a representative posture (i.e., posture P3—a palmar grasp). Superimposed on the position trajectories is the desired target position (continuous horizontal line) within the ±15% envelope (dotted horizontal lines). Gray time window (the same for all four graphs) denotes when the four DoFs were positioned within the ±15% envelope; thus time metric Tc is by definition at the end of such a gray window.

Fig. 3.

Posture matching experiment outcomes. Whiskers denote the standard error of the mean. (a) Performance metrics achieved by subjects S3 and S4. In the bottom panel, bars refer to the minimum posture error (MPE) whereas circles refer to the End-of-trial mean Posture Error (EPE) and were computed only for unsuccessful trials. (b) Completion rate (CR) and completion time (T_c) as a function of the number of controlled degrees of freedom.

The MPE and EPE combined graph [lower panel in Fig. 3(a)] provides interesting insights on the performance of the subjects as well as enlightening the differences between S3 and S4 in CR. Although S4 could successfully complete only a fraction of the trials, his MPE ranged between 1 ± 1.2% and 21 ± 3% with very small variability across postures. This indicates that S4 was close to the spatial requirements of the target envelope (i.e., within 15% RoM) but never satisfied the temporal requirements of having all four DoFs within the envelope for 0.5 s. In unsuccessful trials the EPE was low and ranged between 22 ± 3.5% and 45 ± 0%. In short, S4 was unable to perfectly master but formed postures that were qualitatively close to the targets. Subject S3 showed a different behavior: the MPE was on average lower than S4 but with greater variability across postures and this was mainly due to the postures that were not successfully achieved (cf., Fig. 3(a), bottom panel). In the two worse cases (posture P8 and P9) the MPE was 34 ± 3% for posture P8 and 19 ± 3% for P9. In fact, S3 was able to form many of the target postures but was absolutely unable to reach some others (P8 and P9).

The graph in Fig. 3(b) displays CR and Tc as a function of the number of controlled DoFs (i.e., input channels) for each subject. This graph was computed by removing the lowest performing DoF at each step, for each subject. While the overall accuracy in controlling four DoFs into 12 arbitrary postures was 53 ± 11% for S3 and 28 ± 7% for S4, the performance increased to 65 ± 10% and 42 ± 8% with three controlled DoFs and to 76 ± 8% and 67 ± 7% for two controlled DoFs. Similarly, the time to completion decreased from Tc_S3 = 5.8 ± 0.6 s, Tc_S4 = 5.4 ± 0.9 s with 4 DoFs, to Tc_S3 = 5.1 ± 0.5 s, Tc_S4 = 5.2 ± 0.6 s with 3 DoFs and Tc_S3 = 3.7 ± 0.4 s, Tc_S4 = 3.8 ± 0.5 s with 2 DoFs.

Experiment 2A–B

The temporal graph in Fig. 4(a) shows data from subject S1 depicting the relative controlled DoF positions and the EMG traces for the four muscles, for a representative sinusoid (with frequency 0.13 Hz, mean value of 75% of total span, and range of movement 40%). Subjects S1 and S2 demonstrated significantly improved performance compared to subjects S3 and S4 as assessed by the Wilcoxon rank sum test on the MAE(p ≪ 0.01), RMA (p = 0.0035) and RVA (p = 0.0095) [see Fig. 4(b)].

Fig. 4.

Sinusoid tracking experiment outcomes. (a) EMG signals and relative controlled DoF positions for a representative sinusoid. (b) Differences in performance metrics between subjects S1 and S2 and subjects S3 and S4. Asterisks denote statistically significant differences as computed by Wilcoxon rank sum tests. (c) Confusion matrices of the relative mean activity (RMA) and relative variance activity (RVA) from subjects S1 and S2 (experiment 2B). (d) Confusion matrices of the RMA and RVA from subjects S3 and S4 (experiment 2A).

The MAE was 6.6 ± 0.25 % (mean ± standard error of mean) for subjects S1 and S2 (average over 12 trials) and 16.4 ± 1.9% for subjects S3 and S4 (16 trials). These results demonstrate good tracking abilities when the target sinusoid was slow (<8.3%_RoM/s) and fair performance when fast (≥ 8.3%_RoM/s) using the present interface.

The confusion matrices [Fig. 4(c) and (d)] show the degree of independence of the four DoFs during each sinusoid tracking task in terms of relative mean activity (RMA) or relative variance activity (RVA) averaged across the trials and subjects. The average RMA (depicted by the color of the nondiagonal elements of the confusion matrices) across all sinusoids was 26 ± 3.8% for subjects S1 and S2 and 45 ± 4.4% for subjects S3 and S4. The average RVA was 25 ± 3.7% for subjects S1 and S2 and 36 ± 3.4% for subjects S3 and S4.

The RMA and RVA demonstrate that the neural drive of the nontest muscles could be intentionally attenuated to some extent (on average, the mean value and variance of the nontest DoF was one quarter of the mean value and variance of the test DoF) by the subject when the speed of the target signal was slow, whereas it was not possible at faster signals.

IV. Discussion

The goal of these experiments was to assess whether independent, continuous and simultaneous real-time control of four intrinsic DoFs of the hand was possible by recording the physiologically appropriate extrinsic muscles using intramuscular electrodes. We believe that we have demonstrated that it is indeed possible to simultaneously control four DoFs in real-time using EMG signals from intramuscular electrodes.

In experiment 1 the subjects were asked to form the hand, in real time, into 12 target postures. This experiment was challenging because subjects were required to “undo” unintended movements and command DoF stopping, because all DoFs were needed to match the target posture. Strict performance as measured by the CR varied greatly between subjects and postures and on average subject S3 achieved significantly higher CR than subject S4. While the performance metrics (particularly CR) seem unremarkable, the experiment could be deemed a success when considered in the light of the following physical and biological considerations.

Some of the posture errors during experiment 1 were possibly due to the fact that subjects were not given quantitative visual feedback of the positions of the DoFs of the hand. Instead, subjects qualitatively judged the posture of the hand by looking at it from a limited perspective. Therefore, there were instances when postures of the physical hand looked, and so were judged by the subjects, to be correct (as in the video clip in the supplementary materials), when in fact had more error than was allowed and was considered incorrect when processed. We hypothesize that the posture errors increased due to the lack of full system observability.

The orthopaedic splint that was worn by the normally limbed subjects also caused some confounding issues. This splint helped to produce comfortable muscle contractions, but it also induced subjects to produce small compensatory finger movements and contractions. Every force applied to the splint by one DoF caused slight deflections and movements of the splint itself which induced counterbalancing forces by the other fingers and the wrist. The muscle contractions due to the presence of the splint produced undesired movements of the other DoFs of the robotic hand. It is worth noting that this kind of compensatory contraction would not be induced in the case of an amputated hand, as the splint would not be needed.

The third issue related to muscle coactivation which is an intrinsic property of the motor control system [11]–[17]. In his work on intrinsic hand movements, Birdwell and colleagues showed that the activation of one muscle systematically resulted in coactivity of the neighboring muscles [17]. Although the EMG activity of the intentional contraction can be significantly greater than the unintentional ones, the ratio might vary from subject to subject and between different muscles. Neural coactivations are present that in turn generate significant EMG levels and hence unintended movements in the case of the present human machine interface (HMI). Birdwell showed that light activations of FDP₂ resulted in coactivity of FDP₁ that could be quantified as ~70% relative to the FDP₂ activity. Conversely FDP₁ contractions resulted in reduced coactivity of FDP₂ (5%–10% of FDP₁ activity). Similarly, APL activations resulted in strong coactivity of FPL, (50%–110% of APL activity), whereas FPL contractions produced instead low APL contractions (15% of FPL activity). In Birdwell’s work contractions were limited to 10%–30% of MVC, whereas in this study some postures required ~70% MVC in order to fully close one DoF.

Muscle coactivations played a confounding role in the way the hand performed, especially with target postures that involved alternate full closure/full opening of one or more DoFs. In fact, most of the postures required unnaturally precise and sustained contractions (see position control discussion below) in order to be quantitatively matched (cf. Fig. 1).

Judgment of the correct posture, the presence of the splint and muscle coactivations played different roles in the subjects’ ability to control the hand and potentially explain the differences in performance by subjects S3 and S4, as follows. The low performance by subject S4 may have been a result of poor perspective and judgement of the hand position. Most of subject S4’s postures were close to correct (low MPE and EPE, consistent among postures) but only a few precise enough to be considered correct. The inability of subject S3 to reach postures P8 and P9 could be mainly explained by muscle coactivations and the presence of the splint. In summary, the combined observation of CR, MPE and EPE metrics evidences the successful ability in controlling the hand towards the correct (or close to correct) posture demonstrated by both subjects.

The sinusoid tracking experiment was aimed to address dynamic control of the hand. Results were generally promising and the poor subject performance (high RMA and RVA for subjects S3 and S4) was probably due to the combination of muscle coactivations, the presence of the splint and the difficulty of the task. Coactivations were considerably lower for subjects S1 and S2 who tracked a different, slower, set of sinusoids. It is worth recalling that even in the sound hand single muscle activations do not always result in single DoF movements as the tendons, biomechanical couplings, and neurological couplings affect other movements.

A weakness of the present work is that individuals with amputations did not participate in the study. However, we were constrained by a lack of access to such individuals in the time frame available to us. So we chose to conduct this exploratory study by enrolling intact limbed participants. Nevertheless the outcomes are promising and experiments with individuals with amputations are foreseen. We consider results from the present experiment a real breakthrough towards “natural” dexterous hand control for individuals with upper-limb amputations. It should be noted that the possibility of implementing the present approach in a potential future practical system is bound to the availability of multi-digit hand prostheses [6], [7] and most of all to the technological development of implantable myoelectric sensors (IMES), along with telemetric data and power transfer, e.g., [5]. Indeed only intramuscular EMGs—i.e., free from crosstalk from neighboring muscles—recorded by an implant could be mapped one-by-one to corresponding DoFs in a hand prosthesis in order to replicate the present approach, because obviously, percutaneous fine wires could not be used in a chronic setup. The muscles/compartments targeted in this work were chosen because they articulate DoFs in the hand that are used to perform tasks of daily living. Interestingly, all four muscles/compartments lie in the proximal third of the forearm, so that they could potentially be targeted by IMES and mapped to the corresponding DoFs in a multi-digit hand prosthesis, in a person with transradial amputation. However, the present approach is quite general and different DoFs could be targeted by IMES depending on the needs of a specific person.

The use of position control implemented in the present HMI, as opposed to a velocity controller [25], posed unique challenges for the experimental tasks because it imposed the subjects to maintain the posture within the target envelope by contracting all four muscles. This proved to be a difficult task. We hypothesize that a velocity controller would reduce the difficulty since a constant contraction would not be necessary to hold the posture within the target envelope. While neural coactivations limited the performance obtained by the present HMI, it may be possible to achieve higher independence among DoFs by targeting the extensor and flexor muscles and implementing a differential velocity controller. In fact, most of the movements in the unimpaired hand are the result of a combination of contractions of flexor and extensor muscles and thus it is reasonable to expect greater controllability by targeting more muscles actually involved in the movements (not only the flexors). On the other hand, one could maintain a reduced number of control channels (i.e., implanted sensors) and simply accept the fact that postures and grasps cannot be exact in numbers but qualitatively correct as in the human nature. Qualitatively correct control combined with artificial devices endowed with mechanical intelligence could contribute to solving the challenging goal of restoring the human hand grasping function.

When making an analytic comparison between this and previous works on physiologic control of intrinsic hand movements using peripheral information [1]–[3], [26]–[28] significant differences can be found. Effortless control and continuous prehensile movements are two of the main features that characterize the unimpaired brain–hand connection. Thus it is a well-established argument that both of these features should be provided by a prosthetic hand in order to successfully restore the motor function. However, previous studies either demonstrated proportional control of only one DoF [1], [2] or the control of multiple DoFs using pattern recognition which by definition allows for a finite number of postures/grasps [3], [26]–[28]. Among the works that investigated pattern recognition from surface EMGs, Tenore et al. [26] demonstrated significant classification accuracies (>90%) of ten different hand postures (offline processing) in a transradial amputee; Cipriani and colleagues [27] showed real-time control of seven postures with average accuracy of 79% across five transradial amputees. In the only study that investigated multi-DoF hand postures using pattern recognition of peripheral neural signals, Rossini et al. [3] demonstrated the possibility of classifying four postures (including relax/neutral) by processing offline signals recorded from one amputee using intrafascicular peripheral electrodes in the ulnar and radial nerves. Pattern recognition controllers have demonstrated promising performance and usability but with the critical limitation that the person can only sequentially control a set of predefined postures and thus cannot control arbitrary movements in a continuous fashion. The present HMI potentially overcomes these limits. We argue that although it could demonstrate some limitations in the practical application with amputees, the present approach holds the potential of being superior to any existing finite-states machine, since it mimics the normal motor control. However, while having an HMI that allows for continuous control over multiple DoFs is desirable, it is also true that such continuous control could lead to unstable outputs (i.e., the fingers continuously move drawing current from the battery). For instance, especially in the case of a position control scheme (as the one adopted in this work) the individual would need to maintain a specific contraction in order to keep a specific posture/grip of the hand. This condition is not desirable especially with powerful contractions (>10%–20% MVC) but could be addressed and eventually solved by a velocity control scheme.

Although not strictly with reference to hand movements, other studies relevant to the present work were carried out. These include studies in which intramuscular techniques or the targeted muscle reinnervation (TMR) were deployed in order to control in parallel multiple DoFs of upper limb pros-theses. In particular, both Merrill et al. [18] and Kamavuako et al. [29] demonstrated simultaneous and direct control of hand open/close and wrist pronation/supination (a two-DoF prosthesis) using fine-wire electrodes and physiologically appropriate muscles. TMR is a surgical technique where nerves originally serving the hand/wrist are rerouted and reinnervated to muscles in the chest; this makes available new, independent control sites for surface EMG recording [4]. TMR has proven to be a viable means to regain physiological control of three-DoF prostheses (simultaneous control of hand open/close, wrist pronation/supination and elbow flexion/extension) using pattern recognition [25].

Concluding, to our knowledge this is the first study in which peripheral neural correlates were processed in real-time and used to control four DoFs of a physical hand in a simultaneous and in a very natural way. This step forward was possible by using as control inputs myoelectric signals from those specific muscles that originally articulated the fingers. Besides allowing subjects to achieve continuous simultaneous and independent control over multiple DoFs, this technique enables the subject to perform potentially any arbitrary posture in a physiologically appropriate manner. Though further developments are still required in order to make chronically implantable myoelectric sensors available, we believe that the present approach represents a concrete yet clinically viable alternative to the use of neural electrodes implanted in the peripheral nerves. Interestingly, implanted myoelectric sensors could be combined with the TMR technique in order to naturally restore several of the DoFs lost after a proximal or distal amputation.

Biographies

Christian Cipriani (S’06–M’09–SM’12) received the M.Sc. degree in electronic engineering from the University of Pisa, Pisa, Italy, in 2004, and the Ph.D. in biorobotics from the IMT Institute for Advanced Studies, Lucca, Italy, in 2008.

He is currently an Assistant Professor and Head of the Artificial Hands Laboratory at the BioRobotics Institute, Scuola Superiore Sant’Anna, Pisa. He is the Coordinator and PI of the MY-HAND Project (RBFR10VCLD) funded by the Italian Ministry of Research and of the WAY Project (ICT #288551) funded by the European Commission. He was a Visiting Scientist at the University of Colorado, Denver, Anschutz Medical Campus, in 2012, and he founded a spin-off company, in 2009. His research interests cover mechatronics, controllability and sensory feedback issues of dexterous robotic hands to be used as thought-controlled prostheses.

Dr. Cipriani won the d’Auria Award for prototypes of innovative robotic devices to aid the motor disabled from the Italian Robotics and Automation Association, in 2009. In 2011, he was awarded with an early career grant (FIRB program) by the Italian Ministry of Research and with a Fulbright Research Scholar fellowship.

Jacob L. Segil (M’11) received the B.S. degree in mechanical engineering from the University of Illinois, Urbana–Champaign, IL, USA, in 2008 and the M.S. degree in mechanical engineering from the University of Colorado, Boulder, CO, USA, in 2012. He is currently pursuing the Ph.D. degree in mechanical engineering at the University of Colorado at Boulder.

From 2008 to 2010, he was a Research Engineer in the Center for Bionic Medicine at the Rehabilitation Institute of Chicago (RIC). Currently, he is a Research Assistant in the Biomechatronics Development Laboratory at the University of Colorado, Denver, Anschutz Medical Campus. His interests include mechatronic design and brain machine interfaces, in particular myoelectric control of prosthetic limbs.

J. Alex Birdwell (S’11–M’12) received the B.S. degree in mechanical engineering from the Georgia Institute of Technology, Atlanta, GA, USA, in 2004, and the M.S. and Ph.D. degrees in mechanical engineering from Northwestern University, Evanston, IL, USA, in 2006 and 2012, respectively.

He is currently a Lecturer of mechanical engineering at Northwestern University. His research interests lie in the intersection of robotics and biomechanics and mainly focus on prosthetics, human–machine interactions, and rehabilitation engineering. He was previously a graduate Research Assistant in the Center for Bionic Medicine at the Rehabilitation Institute of Chicago, Chicago, IL, USA. He currently enjoys teaching an array of manufacturing, design, thermodynamics, and experimental methods courses.

Richard F. ff Weir (S’92–M’95) received the B.A. degree in mathematics and a BAI degree in microelectronics and electrical engineering from Trinity College, Dublin, Ireland, in 1983. After working as a control engineer in U.K., he moved to the USA and received the M.S. and Ph.D. degrees in biomedical engineering from Northwestern University, Evanston, IL, USA.

He is currently Director of the Biomechatronics Development Laboratory, University of Colorado Denver Anschutz Medical Campus. He is also a Research Healthcare Scientist for the VA Eastern Colorado Health Care System, Denver VA Medical Center, and holds Research Associate Professor appointments in the Departments of Bioengineering and Physical Medicine and Rehabilitation at the University of Colorado Denver, Anschutz Medical Campus. He specializes in the design and development of advanced artificial hand/arm replacements. His research covers all aspects of the problem ranging from development of neural control interfaces and clinical deployment of these systems, to mechatronic design and development of novel prosthetic components. Current projects involve the development of prosthetic hand/arm controller systems based on implantable myoelectric sensors (IMES) to create a neural interface for the user. He is also conducting research into novel ways to interface with peripheral nerves using optogenetic approaches as well as a number of straightforward mechanical hand and arm component design projects.