The Target Achievement Control Test: Evaluating real-time myoelectric pattern recognition control of a multifunctional upper-limb prosthesis

Abstract

Despite high classification accuracies (~95%) of myoelectric control systems based on pattern recognition, it is unclear how well offline measures translate to real-time closed-loop control. Recently, a real-time virtual test analyzed how well subjects completed motions using a multiple–degree of freedom (DOF) classifier. Although this test provided real-time performance metrics, the required task was oversimplified: motion speeds were normalized and unintended movements were ignored. We included these considerations in a new, more challenging virtual test, the Target Achievement Control (TAC) Test. Users attempted to move a virtual arm into a target posture using myoelectric pattern recognition. Five transradial amputees performed the test with various classifier (one vs. three DOF) and task complexities (one vs. three required motions per posture). No significant difference was found in classification accuracy between the one- and three- DOF classifiers (97.2%±2.0% and 94.1%±3.1%, respectively) (p=0.14). Subjects completed 31% fewer trials in significantly more time using the three-DOF classifier. Subjects took 3.6±0.8 times longer to reach a three-motion posture compared to a one-motion posture. The results highlight the need for closed-loop performance measures and demonstrate that the TAC Test provides a useful and more challenging tool to test real-time pattern recognition performance.

Introduction

Myoelectric control systems based on pattern recognition have been proposed for the next generation of multifunctional upper-limb prostheses (1–3). Ideally, a multifunctional prosthesis will restore functionality to a patient and provide a measurable improvement in their quality of life. Unfortunately, the only validated prosthetic outcomes measure is the Assessment for Capacity of Myoelectric Control (ACMC) (4, 5). This test measures users' ability to perform a series of two-handed tasks and requires a physical prosthesis under volitional control. The Upper Limb Prosthetics Outcome Measures (UPLOM) Group, was formed in 2005 to address the lack of outcomes measures for upper-limb prosthetics (6). In 2009, this group presented findings that identified a wide range of variables that contribute to prosthesis usability. One variable, “Control of the Prosthesis”, included the desire to have a measurement tool that is sensitive enough to differentiate between control schemes and show changes in the ability to control the prosthesis over time (7). The group recognized that the evolution of prosthesis design constitutes a continuum from research, development, clinical work, and ultimately home use. They recommended that a series of tests be used to iteratively test each preceding stage in the continuum. In this contribution, we are working in the development portion of the continuum and are attempting to develop a test which may be used to quantify control algorithms which have previously been researched, prior to the time when it is realistic to implement with physical devices, which may or may not yet exist Specifically, we wished to develop a test to measure the patient’s ability to control pattern recognition systems of varying complexity and comprised of different components.

During pattern recognition control, a computer program identifies an individual’s intended movements by looking at the pattern produced by several channels of surface electromyographic (EMG) signals (8). The pattern is classified and a movement command is sent to the prosthesis. A large focus of pattern recognition research is to provide better EMG decoding through use of various classifiers and feature sets (3, 9–12). The performance of a classifier is commonly assessed by calculating its classification accuracy after all data has been collected. Classification accuracy is the ability of the algorithm to correctly decode users’ movements. Pattern classification techniques, such as linear discriminate analysis (1, 13), fuzzy logic (3, 14), or artificial neural networks (10, 11, 13) commonly achieve offline classification accuracies above 95%.

It is relatively unclear, however, how a pattern classifier’s performance in offline tests translates to its performance in real-time closed-loop control (15). Data is collected while the subject tries to produce a specific motion. After the experiment is over, the data is processed and classification accuracy is calculated as the percent of time the classifier correctly identifies the motion. Therefore classification accuracies are calculated during an open-loop task in which the user has no feedback. At the beginning stages of development, offline accuracies provide useful information without the need for a multifunctional prosthesis. With offline performance established, the need for evaluation tools based on real-time performance becomes more apparent, as it is important to investigate what happens to performance when the user is interacting with the decoded movement.

Virtual environments can provide an alternative setting for evaluating real-time pattern recognition performance (16–19). Through the use of a virtual clothespin task, Hargrove et al. showed that system controllability and functional performance improved when the transient portion of EMG signals was included in classifier training (17). This is of noted importance because including this information may have the opposite effect on offline classification accuracy by lowering the reported performance of the system (17). More recently, Kuiken et al. designed a virtual test, called Motion Test, which examined the clinical robustness and accuracy of pattern recognition control (18) with amputees who had undergone targeted muscle reinnervation (TMR) surgery (20, 21). During this test, subjects were instructed to follow prompts for a movement and observe a virtual prosthesis that decoded their movements. Subjects were told to maintain their muscle contractions until the virtual prosthesis moved through its full range of motion (18). Previous Motion Test results suggest that the reinnervated muscles of TMR amputees can produce sufficient EMG information for real-time pattern recognition control (18). Although this test provides real-time performance metrics, the required task is oversimplified; motion speeds are normalized and unintended movements (i.e. misclassifications) are ignored.

We included these considerations (i.e. motion speeds and misclassifications) in a new more challenging virtual test, the Target Achievement Control (TAC) Test. This test evaluated users’ control and positioning of a multifunctional prosthesis. Users were instructed to move a virtual prosthesis into a target posture (Figure 1) and maintain the posture for a period of time (i.e. 2 seconds). If the user overshot the target posture or produced unnecessary movements (either through volitional control or motion misclassifications), these movements needed to be corrected to achieve success.

Figure 1.

Target Achievement Control (TAC) Test. Subjects moved a multifunctional virtual prosthesis into a target posture. The virtual hand turned green when the target was reached within the acceptable tolerances (± 5 degrees for each degree of freedom). Pictures illustrate starting and ending positions for successful trials. (A) Example trial from Conditions 1 and 2 requiring one motion to reach the target posture (e.g. wrist flexion). (B) Example trial from Condition 3 requiring three motions to reach the target posture (e.g. wrist flexion, wrist supination, and hand close).

In this study, individuals with a transradial amputation controlled a virtual prosthesis using myoelectric pattern recognition with proportional control. To illustrate the flexibility of the new virtual performance test, subjects performed the TAC Test with two classifier complexities (a one–degree of freedom classifier and a three–degree of freedom classifier) and two task complexities (one or three motions required to achieve target posture success). The results showed that the TAC Test provided valuable information about users’ myoelectric control and the pattern recognition control algorithms that could not be obtained with existing performance measures (e.g. offline classification accuracy) or existing real-time virtual performance tests (e.g. Motion Test).

Methods

Subjects

Five individuals with a transradial (TR) amputation participated in this study (Table 1). The experimental protocol was approved by the Northwestern University Institutional Review Board and all subjects gave written informed consent to participate.

Table 1.

Transradial subject demographics

Subject	Age	Amputation Arm	Arm Tested	Time since Amputation	Type of Prosthesis Used
TR1	53	Right	Right	20 years	Myoelectric
TR2	62	Right	Right	25 years	Myoelectric
TR3	55	Bilateral	Right	32 years	Body-Powered
TR4	24	Left	Left	9 months	Body-Powered
TR5	32	Bilateral	Right	3 years	Body-Powered

EMG and Pattern Recognition Configuration

Six self-adhesive silver/silver chloride bipolar surface electrode (Noraxon Dual electrodes) pairs were used to record muscle activity. The electrode pairs had a 1 cm diameter circular conductive area and an inter-electrode distance of 2 cm. Four electrode pairs were placed in a ring at the proximal portion of the residual forearm around the apex of the muscle bulge (2 to 3 cm distal to the elbow crease) and the other two electrode pairs were positioned on the distal end of the residual limb. For subjects with short residual limbs, all six electrodes were placed in the ring at the proximal portion of the forearm. The EMG signals were amplified, sampled at a frequency of 1 kHz, and high-pass filtered at 20 Hz to reduce motion artifact.

Subjects trained the system to recognize seven motion classes (wrist flexion, wrist extension, wrist supination, wrist pronation, hand open, one hand grasp, and no movement). For training of the pattern recognition system, subjects were prompted with a demonstration of each movement and asked to perform the movement at a comfortable and consistent level of effort. Each contraction was held for 3 s and repeated eight times. The data were split into two groups with 12 s of data used to train a linear discriminate analysis (LDA) classifier and 12 s of data used to test the classifier. The pattern recognition system segmented data from all EMG channels into a series of 150 ms analysis windows with a 50 ms window increment. Four time-domain features (mean absolute value, number of zero crossings, waveform length, and number of slope sign changes) were extracted from each analysis window. With this classifier, only one class decision was made at a time (i.e. sequential control). This pattern recognition scheme has been previously described (1) and has shown to produce effective real-time control (18, 22). After the classifier was trained, it was used to predict user commands and control a virtual prosthesis in real time. Classification accuracy was assessed offline by dividing the number of correct class decisions by the total number of class decisions.

For this experiment, the proportional movement speed was calculated by averaging the mean absolute values (MAV) of all channels, k, of EMG signals for a given data window and multiplying by a class gain factor, G (15, 23).

$${\mathit{Speed}}_i=G_i\left(\frac{1}{N}\sum _{k=1}^N{\mathit{MAV}}_k\right)$$ (1)

Desired speed gains were configured for each class such that subjects could achieve full dynamic range where the maximum EMG amplitude corresponded to 100 degrees per second. Subjects practiced in the virtual environment for 5 to 10 min. prior to testing.

Target Achievement Control Test

A target posture and a virtual prosthesis that responded to classifier output were displayed on a screen (Figure 1). Subjects were instructed to move the virtual prosthesis, which started from a non-neutral position, to a neutral target posture. The neutral position was zero degrees of wrist flexion/extension and zero degrees of wrist rotation (see Figure 1, successful trial end) To provide visual feedback, the virtual hand turned green when it was within an acceptable tolerance of the target (±5 degrees for each degree of freedom) (Table 2). Tests were completed more quickly if subjects only produced the motion(s) necessary to reach the target. If a subject overshot the target posture or produced unnecessary movements, he/she had to correct for those motions to achieve success. Trials ended successfully when subjects were able to keep the virtual prosthesis in the target for 2 s. Target postures were never at the end of DOF ranges which ensured controlled stopping and dwelling within the target posture as part of the required task. Trials ended unsuccessfully if subjects were unable to achieve and maintain the target posture by the specified trial timeout. Three conditions were tested in this study:

Table 2.

Target Achievement Control Test configurable parameters

Parameter	Description	Study Setting
Test Complexity	Number of motions required to reach the target posture.	1 (C1 & C2) 3 (C3)
Movement Distance	Distance between the initial position of the virtual hand and the target posture for each tested motion. Larger or smaller distances can be used to test gross or fine motor control.	75 degrees
Target Width	Acceptable tolerance for reaching the target posture. Smaller target widths lead to more challenging trials.	± 5 degrees
Dwell Time	Length of time the virtual prosthesis has to continuously remain in the target posture for the trial to be considered successful.	2 s
Trial Timeout	Length of time in which trial must be completed. If the timeout is reached without success, the trial is considered failed.	15 s (C1 & C2) 45 s (C3)

Condition 1

Subjects controlled a one–degree of freedom virtual prosthesis and performed the TAC Test with one required motion per trial. Each degree of freedom was tested separately. For wrist rotation, only the data for wrist supination, wrist pronation, and no movement were used to build and test the LDA classifier. The target posture required subjects to either supinate or pronate the virtual wrist across a movement distance of 75 degrees to achieve success. The protocol was repeated for wrist flexion/extension and hand open/close. The order that the one-DOF classifiers were presented to the subject was randomized. For each one-degree of freedom classifier, subjects performed four sets of the TAC Test; each set consisted of two repetitions of each target posture (two postures) with a trial timeout of 15 s. Condition 1 consisted of a total of 48 trials.

Condition 2

Subjects controlled a three–degree of freedom virtual prosthesis and performed the TAC Test with one required motion per trial. Data for all seven motions were used to build and test the LDA classifier. Similar to Condition 1, the target posture only required subjects to perform one motion across a movement distance of 75 degrees to achieve success. Unlike Condition 1, all three degrees of freedom were active during each trial. For example, if a subject was trying to pronate his/her wrist and the hand closed, he/she needed to re-open the hand before achieving the target posture. Subjects performed four sets of the test; each set consisted of two repetitions of each target posture (six postures) with a trial timeout of 15 s. Condition 2 consisted of a total of 48 trials.

Condition 3

Subjects controlled a three–degree of freedom virtual prosthesis and performed the TAC Test with three required motions per trial. Similar to Condition 2, data for all motions were used to build and test the LDA classifier. Unlike Condition 2, target postures required subjects to perform three motions, such as wrist flexion, wrist supination, and hand open, to achieve success. Each posture required moving the virtual prosthesis across a distance of 75 degrees for each required motion. Therefore subjects had to move the virtual prosthesis over a total distance of 225 degrees. Subjects performed four sets of the test; each set consisted of one repetition of each target posture (eight postures). Since the pattern recognition algorithm used in this study allowed only sequential motions, the trial timeout for Condition 3 was 45 s. Condition 3 consisted of a total of 32 trials.

Prior to testing, subjects were given at least 5 min. to familiarize themselves with each condition. Conditions were presented in a randomized order. The first test set of each condition was used as practice, and subsequent sets were used for data analysis. The effects of classifier complexity were analyzed by comparing Conditions 1 and 2, and the effects of task complexity were analyzed by comparing Conditions 2 and 3.

TAC Test performance metrics included completion time, completion rate, and path efficiency. Completion time was the time from trial start to the successful achievement of the target posture, not including the 2 s dwell time. Completion rate was the percentage of successfully completed postures in a set of trials. Path efficiency was calculated as the shortest path to the target divided by the total distance traveled by the virtual hand (24). Therefore, a trial with path efficiency equal to 100% indicated that the subject was able to move the virtual prosthesis directly into the target posture and stop within the acceptable tolerance. Completion time and path efficiency were only reported for successful trials.

Statistical Analysis

We performed a paired t-test to assess the statistical difference between classification accuracy, completion rate, completion time, and path efficiency across the two levels of classifier and task complexities.

Results

Classifier Complexity (Comparison of Conditions 1 and 2)

Classification accuracy was not significantly different between the one- and three- DOF classifiers (p = 0.14). Average classification accuracy was 97.2% ± 2.0% (mean ± standard deviation) across all one–degree of freedom classifiers (Condition 1) and 94.1% ± 3.1% across all three–degree of freedom classifier (Condition 2).

When the TAC Test required only one motion per posture, subjects completed significantly more trials and completed them significantly faster while using the one–DOF classifier compared to using the three–DOF classifier (p = 0.002 for completion rate and p < 0.001 for completion time) (Figure 2, Table 3). Figure 3 displays the position and decision history of an example trial using the three–DOF classifier. Path efficiency measures demonstrated a similar trend of significantly decreased performance with the three- compared to the one–DOF classifier (p = 0.03).

Figure 2.

Average completion rate curves for all three conditions. Solid line indicates performance during trials that required only one motion per posture using a one–degree of freedom classifier (Condition 1). Dashed line indicates performance during trials that required only one motion per posture using a three–degree of freedom classifier (Condition 2). Dotted line indicates performance during trials that required three motions per posture using a three–degree of freedom classifier (Condition 3). Shaded regions represent ± 1 standard error.

Table 3.

TAC Test Performance Measures by Condition

Condition	Classifier Complexity	Task Complexity	Classification Accuracy, %	Completion Rate, %	Completion Time, s	Path Efficiency, %
1	1-DOF	1-motion	97.2 ± 2.0*	99.4 ± 1.2*	2.9 ± 1.0*	92.8 ± 3.9*
2	3-DOF	1-motion	94.1 ± 3.1*	68.9 ± 9.3*^	5.6 ± 0.9*^	81.1 ± 5.0*^
3	3-DOF	3-motions	94.1 ± 3.1	92.1 ± 7.6^	20.1 ± 4.0^	54.7 ± 11.1^

^*Paired t-test indicates significant difference between Condition 1 and Condition 2 (p < 0.05).

^{^}Paired t-test indicates significant difference between Condition 2 and Condition 3 (p < 0.05).

Figure 3.

Position and decision history during an example TAC Test trial requiring one motion to reach a Condition 2 target posture. A three–degree of freedom classifier is used. The virtual prosthesis began in 75 degrees of wrist extension, 0 degrees wrist rotation, and the hand 50% closed. The user had to flex the wrist to reach the target posture (0 degrees flexion/extension, 0 degrees wrist rotation, and the hand 50% closed). Gray bars indicate the target position for each degree of freedom. Since the TAC Test required all degrees of freedom to match the target position, the subject had to correct for any misclassifications (e.g. wrist pronation). The virtual arm reached the target position at 5.3 s (indicated by T^*). The trial ended at 7.3 s after the subject was able to remain in the target posture for 2 s.

Task Complexity (Comparison of Conditions 2 and 3)

When the TAC Test required subjects to perform three motions to achieve each posture (Condition 3), subjects completed significantly more trials (p = 0.03) in a significantly longer time (p = 0.001) compared to performing only one motion (Condition 2) (Figure 2, Table 3). Note that the trial timeout length for Condition 3 was three times that of Condition 2. Figure 4 displays the position and decision history of a subject using the three–DOF classifier to reach a posture that required three motions. The average completion time for achieving a three-motion posture was 3.6 ± 0.8 times longer than the average completion time for a one-motion posture. The average path efficiency for the three-motion posture was significantly lower than that for the one-motion posture (p = 0.01).

Figure 4.

Position and decision history during an example TAC Test trial requiring three motions to reach a Condition 3 target posture. The virtual prosthesis began in 75 degrees of wrist flexion, 75 degrees of wrist supination, and the hand 25% closed. The user had to extend and pronate the wrist and close the hand to reach the target posture (0 degrees flexion/extension, 0 degrees wrist rotation, and 75 degrees hand open/close). Gray bars indicate the target position for each degree of freedom. The virtual arm reached the target position at 18.2 s (indicated by T^*). The trial ended at 20.2 s after the subject was able to remain in the target posture for 2 s.

Discussion

We investigated subject performance with various classifier and task complexities as a means of highlighting the TAC Test. The existing offline measure of classification accuracy is a limited metric of control due to a ceiling effect. Classification accuracy is bounded by a maximum value of 100%, with pattern recognition algorithms commonly reporting accuracies above 95%. The result of decades of research into classifier types and feature sets are very minimal increases, if any, in classification accuracy, and it is unclear how these changes relate to controllability. Existing virtual performance measures, such as the Motion Test, are oversimplified. The Motion Test prompted subjects to perform one motion until the virtual prosthesis moved through its full range of motion (18). In a previous study, individuals with a transradial amputation controlled 11 motions of a virtual prosthesis with an average classification accuracy of 79% ± 11% (22). Subjects successfully completed 72% of the Motion Test trials (22). In the current study, subjects who performed the TAC Test controlled only 7 motions of a virtual prosthesis (Condition 2) with an average classification accuracy of 94% ± 3.1%. Even with fewer classes and much higher classification accuracy, subjects successfully completed only 69% of the TAC Test trials, highlighting the need for closed-loop performance measures. The TAC Test is challenging because subjects are required to “undo” unintended movements and command DOF stopping, as all degrees of freedom needed to match the target posture. Also unlike the Motion test, the TAC Test allowed subjects to move the virtual prosthesis at a slow or fast rate, depending on the intensity of their muscle contraction.

The TAC Test does not exhibit similar ceiling effects because a wide range of testing difficulties can be achieved by modifying test parameters (Table 2). For example, if a subject was able to achieve a 100% completion rate with, a ±10 degree tolerance on the target posture, the experimenter/clinician can reduce the tolerance to ±5 degrees to make the test more difficult. In the current study, subjects were asked to position the virtual arm into postures that required either one or three movement(s). An interesting observation was that subjects did not seem to be as affected by misclassifications at the beginning of movement while attempting to achieve a three-motion posture compared to a one-motion posture. While performing the TAC Test with one required motion per posture, subjects would often correct unintended movements as they happened (Figure 3). During trials that required three motions per posture, many subjects did not correct movement misclassifications right away, but rather waited until they were closer to the target posture to correct the movements as needed (Figure 4). In this case, misclassifications may actually have helped complete the motion. It is also possible that subjects were unable to tell if the virtual hand was at the target in one degree of freedom before the other degrees of freedom were also close to their target positions. These observations are not possible with other existing virtual performance measures. Similar observations are harder to track while subjects are using a physical prosthesis since current physical prostheses do not include position tracking.

In addition to testing subject performance, the TAC Test provided a good environment for subjects to practice pattern recognition control. To succeed in the test, subjects needed to be able to plan their movements and produce repeatable muscle patterns. Movement timing and sequential control were other key pattern recognition concepts. Users needed to control their muscle contraction length and intensity to properly position the virtual prosthesis and relax their muscles without eliciting another motion in order to keep the virtual arm at the target. Because the pattern recognition algorithm used in the current study only allowed for sequential control, users needed to perform only one motion at a time. Since the TAC Test is not dependent on the type of control, algorithms that provide simultaneous and proportional control (25) can be tested within this virtual environment test if they prove beneficial. The variable configuration (movement distance, time limit, acceptable tolerance, etc) (Table 2) allows the testing of users with different performance levels while still engaging users and maintaining motivation.

The TAC Test must not be confused for a validated upper limb prosthesis usability outcome measure. The ability to control a prosthesis is only one important component in a patients overall ability to use a prosthesis which may include many other variables such as terminal device type, functional level, motivation, and level of therapy (7). The TAC Test was developed to specifically test control algorithms so that differences in control strategies may be measured and compared. This aligns with the findings from UPLOM’s critical recommendation that the information to be captured is specific to the area within which it is being tested. We believe that the test captures important information about the control algorithms being tested and assume that more that systems which score higher in the TAC test will be more controllable and ultimately be more usable. Future work needs to be completed to test that assumption.

One limitation of the TAC Test is that subjects interact with a virtual, not a physical, environment. Although the goal of developing this virtual test was not to completely replicate the physical environment, it is important to acknowledge their differences. User control and performance may differ between these two environments because the virtual environment does not model the prosthesis inertia. Wearing a physical prosthesis may alter the way individuals contract their muscles. Supporting the prosthetic weight also may affect how quickly their muscles fatigue. During physical prosthesis control, it is not only position, but force and acceleration that matter. While performing the TAC Test, subjects can successfully complete trials even with large terminal decelerations (e.g. they can stop abruptly in the target posture). Large terminal decelerations with a physical prosthesis may cause unwanted interactions (e.g. placing a cup down too fast may cause the liquid to spill). Finally, to make the prosthesis usable for the patient, significant therapy would still be required for pattern recognition systems that prove to be controllable in a virtual environment.

Conclusions

Although fundamental differences exist between the virtual and physical environment, the TAC Test provides a good platform for pattern recognition control practice and testing. The virtual test provides real-time performance measures in a variety of testing settings. Current results demonstrate that significant online differences can be seen even when no significant offline differences exist. Therefore it provides more control information in the continuum between the offline measure of classification accuracy and the full setup necessary for physical prosthesis testing.

Abbreviations

DOF

degree of freedom

EMG

electromyographic

LDA

linear discriminate analysis

TAC

target achievement control

TMR

targeted muscle reinnervation