Concurrent surface electromyography and force myography classification during times of prosthetic socket shift and user fatigue

Abstract

Objective

Surface electromyography has been a long-standing source of signals for control of powered prosthetic devices. By contrast, force myography is a more recent alternative to surface electromyography that has the potential to enhance reliability and avoid operational challenges of surface electromyography during use. In this paper, we report on experiments conducted to assess improvements in classification of surface electromyography signals through the addition of collocated force myography consisting of piezo-resistive sensors.

Methods

Force sensors detect intrasocket pressure changes upon muscle activation due to changes in muscle volume during activities of daily living. A heterogeneous sensor configuration with four surface electromyography–force myography pairs was investigated as a control input for a powered upper limb prosthetic. Training of two different multilevel neural perceptron networks was employed during classification and trained on data gathered during experiments simulating socket shift and muscle fatigue.

Results

Results indicate that intrasocket pressure data used in conjunction with surface EMG data can improve classification of human intent and control of a powered prosthetic device compared to traditional, surface electromyography only systems.

Significance

Additional sensors lead to significantly better signal classification during times of user fatigue, poor socket fit, as well as radial and ulnar wrist deviation. Results from experimentally obtained training data sets are presented.

Introduction

Current state-of-art surface myoelectrographic (SEMG) sensors allow amputees to control powered prosthetic or robotic devices. These sensors determine muscle activation in a user’s residual limb by sensing electrical potential change.¹ It has been extensively reported that amputees have regained some lost functionality through the use of multiple degrees of freedom (DOF) of an upper-limb prosthetic.² After the initial learning curve to use the prosthetic, some users report being able to successfully operate robotic hands which include dexterous digits, thumbs, and wrist rotation. Although current powered prosthetic devices provide sufficient dexterity to open doors, grasp glasses and bottles, and carry grocery bags,³ use of these devices continues to be limited outside of the clinical laboratory. Research is underway that will address human–robot interaction challenges and improve control of these devices.

Many prosthetic devices offer users control over only a single DOF, thus reducing operational speed and increasing task completion length. An example of such a task used in clinical settings is to move a number of small foam balls between two boxes, as described in literature.^2,4,5 In this case, it was reported that the time needed to choose the desired action of the powered prosthetic device, and switch between degrees of freedom, comprises a significant portion of the overall duration of the task.⁶ And for a multi-DOF prosthetic device, a user is typically tasked with a tedious switching burden to control one-DOF-at-a-time. To improve usability, Pilarski et al.⁶ showed that automatic DOFswitching could be learned by a control system using an Actor Critic Model with data collected from a SEMG system. This method predicts which DOF a user is likely to control next through a reinforcement learning algorithm. Improvements in simulated tasks of daily life were reported, in particular task completion times were reduced by approximately 14%.⁶ In this study, however, the user was only allowed to quickly move between the relatively gross movements of the elbow and wrist and was only able to successfully function with two DOFs selected at any given time.

Therefore, it is of considerable interest to expand this work to finer and more dexterous movements. However, practical limitations of SEMG sensing technology have often been cited as major challenges for generalizing this approach to SEMG arrays. These limitations include noise and signal degradation over time depending on linear distances along the skin surface above the muscle to be sensed,^7–9 and differences between limb poses during classification and training data sets.¹⁰ User fatigue and sweat, perspiration within the socket, can also cause a degraded EMG signal.^11–13 A method to compensate for signal losses due to sweat was studied by Tomasini.¹⁴ EMG sensors make use of the subject’s skin as a common ground. This shared ground can be highly variable and cause ground-loops; compensation and removal of the DC Offset was found to be possible. Signal processing and EMG signal classification for control purposes is still an active research area, including spatial filtering¹⁵ and methods of preprocessing data.¹⁶ Work in the areas of Neural Networks,^17,18 Gaussian Mixture Models,^19,20 and other approaches have all produced incremental improvements in signal classification.^21–24

Relatively few studies have examined the addition of other sensor input modalities to control upper limb prosthetics. During training, Fougner et al. used accelerometer and an SEMG sensor to help classify grip type.²⁵ More recent studies interfaced an accelerometer with EMG sensors.^26,27 Others have proposed force sensors placed in contact the skin used to detect the changes in force and pressure within the socket due to volumetric changes of the forearm. This technique also known as force myography (FMG),²⁸ residual kinetic imaging (RKI),²⁹ and muscle pressure mapping (MPM)³⁰ shows promise in providing an alternate and additional input for signal classification. Craelius et al. referred to this method as residual kinetic imaging in 1999.^29,31,32 Separately, Phillips described a device with 32 pressure sensors, allowing a pair of lower-arm (transradial) amputees to open and close a simulated prosthetic hand. Forces measured at the interface of the socket and skin surface of the residual limb were used as control inputs.²⁹ More recently, Radmand reported making use of a simulated prosthetic socket with 126 pressure sensors to classify eight hand motion classes. They called the technique “Muscle Pressure Mapping” (MPM).³³ Fit of the prosthetic socket has also been verified using “intrasocket forces”³⁴ and as a means to estimate grip force.²⁸

However, few works have taken multiple sensor input modalities as control inputs into account. Fougner et al.’s use of accelerometers and EMG signals is mentioned above. Time-domain features provided grip type information when combined with linear discriminant analysis during arm motions made by the subject; resting in a neutral position, overhead, and outstretched to one side.²⁵ Thus, prosthetic socket shift as the subject completed simulated activities of daily life was said to be taken in to account by the classifier. Force sensors were integrated with SEMG to “address force induced artifacts when predicting grip-pressure”, by Fougner,³⁵ and showed promising results. Recently, data from an accelerometer were fused with the data from 16 EMG sensors, initially described in Assad et al.²⁶ and followed up in Wolf et al.²⁷ Control signals for a mobile robot, “stop”, “turn”, “go”, were the general focus of the study, but the authors were able to map specific arm gestures and move an actuated robotic gripper.²⁷ Applications in prosthetic control were not explicitly explored. Practical discussions and considerations for fusing accelerometer and EMG data were presented by Radmand et al.³⁰ The clinically impractical needs of training classifiers in a number of possible arm positions is discussed, referred to as “Dynamic training”, and the otherwise very lengthy time requirement for training was able to be minimized somewhat.

In this paper, we report on the use of a combination of SEMG and FMG) sensors along side a neural network classifier to explore classification of data during times of non-ideal prosthetic socket placement due to fatigue, arm position or socket shift. A custom socket was created and fit with both SEMG and FMG activated during flexion and extension of the forearm. In our previous work, we reported on the strong correlation between single pairs of SEMG and pressure sensors,³⁶ which suggests that Force Myography is a good complement to SEMG, and may help with correcting its signal degradation over time. Other work has also shown that it is possible to utilize FMG data in conjunction with grip-selection interfaces.³⁷ This paper details expanded studies of SEMG and FMG data fusion, utilizing a simplified multilayer perceptron feedforward neural network, experimentation with four SEMG and four FMG collocated sensors as a user performs flexion and extension motions of their dominant wrist. Testing using only FMG data is beyond the scope of this study, and proof-of-principle results can be found in prior work^36,37 and in the literature as mentioned above. Results show that classification results can be improved by at least 3% for most cases, 8% when classifying radial and ulnar deviation, and 35% when a socket is shifted due to normal activities of daily living.

Methods

Our previous study³⁶ correlated volumetric changes of the carpi flexor and carpi extensor muscles of the subject’s forearm, which then induced a detectable pressure as applied to a sensor, to “ideal” locations of the SEMG electrodes and “non-ideal” locations of the electrodes. Linear offset, as measured across the surface of the skin, has been shown to affect and degrade signal integrity.^7,8 Placement of the SEMG sensors in “ideal” locations on the flexor and extensor muscle bodies was determined by a physical therapist in order to achieve high-quality EMG data.

Flexion and extension actions were made by the subject and data were recorded. The socket and sensor pair was then manually moved to “non-ideal” locations 1 cm and 2 cm away from the previously determined “ideal” location as measured along the surface of the subject’s skin in each of the four cardinal directions. The flexion and extension movements were then repeated, data were collected, and a signal classification schema built. This proof of concept study has been repeated, expanded, and results and discussion are seen below.

Experimental protocol

Experimental data from a single right-hand dominant, healthy-limbed subject is reported. The subject was informed of the test procedures, which were approved by the local ethics review committee (National Science Foundation NRI Grant #IIS-1208623), and written consent was given by the test subject. In the type of tests conducted, the hardware for sensor housings must be fabricated specifically for each individual subject using 3D scanning of their forearm and 3D printing.

Four experiments were conducted during initial experimentation. These experiments made use of four SEMG sensors attached to the subject’s dominant forearm in order to isolate activity of their flexor and extensor carpi radialis and ulnaris muscles. Four sensor housings containing a piezo-electric force sensor in contact with the surface of an SEMG sensor are contained within a simulated socket described below in the Materials: Socket and sensor housings Section. The sensor placement referenced here as the “ideal position” can be seen below in Figure 1(a) and (b). A metronome provided auditory cues for the subject to perform the defined action at a rate of 40 bpm unless otherwise noted. All SEMG sensors were placed parallel to the muscle body during initial placement.

Figure 1.

(a) Subject wearing SEMG and FMG collocated sensors. (b) Subject wearing simulated prosthetic socket and sensors.

Simulating socket shift

During the first experiment, the subject sat in a chair and rested their dominant arm on the chair’s arm rest in a comfortable position. The subject extended their wrist at least 10 times and returned to a neutral, resting position after each hand motion. Next, the subject was instructed to flex their wrist at least 10 times, returning to a neutral, resting position as before. After collecting data of these initial movements, the SEMG-FMG sensors and housing were rotated or shifted away from this “ideal position”. Sensor orientation parallel to the muscle body was maintained despite this socket shift.

By moving the sensor housing, we simulated the shifting of a user’s prosthetic socket during activities of daily life. The sensor housing was relocated into four separate locations, displaced from the starting, ideal position, by 1 cm. This procedure was initially introduced in our prior study³⁶ and nomenclature has been updated for clarity. Repositioning of the sensor housings occurred in four directions referred to as “lateral”, or rotated counter clockwise when viewing the forearm as if one were the subject, “medial”, or rotated clockwise when viewing the forearm as if one were the subject, “proximal”, or shifted towards the elbow, and “distal”, or shifted towards the user’s hand. The experiment was then repeated, offsetting the sensors and sensor housing 2 cm from the “ideal position”. All sensor housings were rotated or shifted in the same direction during each of the individual experiments, clockwise, counter clockwise, distally or proximally. The subject repeated extension and flexion motions at least 10 times each, in each of the eight offset positions.

Data and further discussion of experimental results are presented in Results and Discussion Sections.

Simulating arm positions during activities of daily living

Additionally, the subject repeated flexion and extension motions in several arm configurations while standing. These arm configurations included positioning the shoulder laterally across the body (adduction), 90° abduction, 135° of shoulder flexion in the sagittal plane, i.e. with the hand at approximately the head level, and 45° shoulder flexion in the sagital plane, i.e. with the hand at approximately the waist level. Data were collected with the socket in the “ideal” position. Figure 2(a) to (d) illustrates the basic arm positions used to during experimentation while the subject was standing.

Figure 2.

Arm positions used while gathering data while simulating arm positions found during activities of daily living. (a) Arm 45° shoulder flexion in the sagital plane (the hand approximately at waist-level). (b) Arm 135° of shoulder flexion in the sagittal plane (hand at approximately head-level). (c) Arm out, away from the body (90° abduction). (d) Arm laterally across the body (adduction).

SEMG and FMG while fatigued

To create forearm muscle fatigue, the subject was instructed to apply force to a sensorized-rectangular piece of plastic with their dominant hand in a “key pinch” grip. A single piezeoresistive sensor was mounted between this piece of plastic and a solid surface with cellophane tape. An initial reading of applied force was taken. Following this initial reading, the subject squeezed a rubber ball covered in felt, i.e. a tennis ball, 10 times at a frequency of 60 bpm. The metronome was used to provide auditory cues for a consistent squeeze rate. The subject then squeezed the force plate as before, providing a reading. This procedure of squeezing a tennis ball and immediately providing a force output measurement was repeated until the subject’s output force measurement was 80% of the initial reading. All readings were taken with the socket in the “ideal position” as defined above.

Recording radial and ulnar deviation of the hand

The subject was seated comfortably in a chair as described before in our previous work.³⁶ The subject sat, rested their arm in a comfortable position and positioned their hand in a neutral position with their thumb pointing “up”. The socket and sensor housing was placed in the “ideal” location on the arm above the forearm muscles. The subject was instructed to deviate their wrist and hand in the ulnar direction, towards the “pinky” finger, repeating this motion at least 10 times. The subject was instructed to return their hand to a neutral, resting position after each motion. The subject was then instructed to deviate their hand in the radial direction, towards the thumb 10 times, returning to a resting neutral position following each motion.

This experiment is meant to capture data of the hand performing a simulated “hammering” or “dart throwing” motion, an activity common during daily life.

Materials

For this study, we have prototyped a heterogeneous sensory input system to control a powered prosthetic device based around four SEMG sensors and four colocated piezo-resistive force sensors. Activation of a desired control input is via excitation of the user’s forearm muscles and the resulting increased intra-socket force. This proof of principle powered prosthetic control system is currently under development in the author’s laboratory.

Socket and sensor housings

Initial work correlating surface EMG and intra-socket pressure made use of a simplified sensor housing which described in Sanford et al.³⁶ and can be seen in Figure 3(a) and (b). This setup allowed a single surface EMG and single pressure sensor to be co-located above the subject’s forearm muscle, and easily relocated as part of that previous experiment. An expanded system, including housings for an opposing pair of pressure sensors was demonstrated in Sanford et al.³⁷ This system continued to make use of piezo-resistive force sensors, using two opposing Flexiforce A201 sensors. This system allowed a user access to two input modalities, flexion and extension of their dominant hand, by sensing changes of intra-socket pressure in a simulated prosthetic socket. Figure 1(a) and 1(b) illustrates the basic placement of the sensors above the muscle bodies activated during gross flexion and extension movements of the subject’s dominant hand.

Figure 3.

Proof of Principle SEMG and colocated piezoresistive force sensor housing. (a) Top and (b) side views. The SEMG sensor can be seen (dark grey), with the collocated force sensor (light grey) attached above it, but under the cantilever.

The system used in this work expands on previous prototypes and includes four Delsys Bagnoli EMG sensors and four collocated FlexiForce A201 model piezo-resistive force sensors. These sensors were positioned around the circumference of the subject’s arm as mentioned in the Methods: Experimental protocol Section. Sampling of EMG data occurred at 2.4 kHz. The four FMG sensors were mounted above the EMG sensors, in contact with the surface of the EMG sensors and the sensor housing. Figure 1(a) illustrates the placement of one of the four pairs of EMG and Force sensors.

Prior to creation of the sensor housings, four EMG sensors were placed above the flexor and extensor muscles of the subject’s dominant forearm. These locations were marked on the subject’s skin. A three-dimensional model of the subject’s dominant arm was then created, using a 3DMD Flex4³⁸ three-dimensional scanning system to provide a scan of the subject’s arm and CAD software. A custom socket was created from this model and sensor housings were created above the marked EMG sensor locations. The socket was 3-D printed using ABS plastic.

Data acquisition

A Delsys Bagnoli EMG system, National Instruments DAQ, and a custom circuit including an Arduino Micro and LabVIEW program were used to gather data for this experiment. The Delsys Bagnoli 16 channel EMG system was directly connected to an NI (National Instruments) USB-6210 DAQ for EMG data acquisition. A custom voltage divider circuit was created to gather pressure data from the four piezoresistive sensors, making use of an Arduino Micro micro-controller. This voltage divider circuit was also connected to the NI USB-6210 DAQ. In conjunction to the mentioned hardware, a custom LabVIEW VI program was written. This program allows users to start and stop data gathering, change data sampling rates, view data in real-time in a graphical display, and output data to TDMS file formats. These TDMS files are later processed and classified and are described in Materials: Classification algorithm.

Classification algorithm

A custom Matlab program was created to process the gathered data, classify that data using a feed-forward neural network, and determine the motion of the subject’s hand. Data acquired using the system described in 3.2 was converted from TDMS file format to CSV for processing.

Then, EMG and FMG data were classified using the Matlab Neural-Network Toolbox in a multi-layer perceptron feed-forward configuration. SEMG parameters described by Hudgins et al. are used as network inputs along with piezo-resistive pressure sensor data.³⁹ These inputs to the network were the moving average of the absolute value of the SEMG signal, the derivative of the absolute value of the SEMG data, and the absolute value of the pressure data. The time vectors of SEMG data and pressure data are denoted as r_t and p_t, respectively. ${\stackrel{\wedge }{r}}_t$ is defined as the magnitude of the time vector of SEMG data, as seen in equation (1).

$${\stackrel{\wedge }{r}}_t=|r_t|$$ (1)

The Central Moving Average of the absolute value of the SEMG signal is referred to as MA and has window size denoted, ω₁. A central moving average computation was performed as shown in equation (2).

$${\tau }_1={\mathrm{MA}}_{{\stackrel{\wedge }{r}}_t}=\frac{1}{{\omega }_1}{\stackrel{\wedge }{r}}_{(t-{\omega }_1)}+\frac{1}{{\omega }_1}{\stackrel{\wedge }{r}}_{(t-{\omega }_1-1)}+\frac{1}{{\omega }_1}{\stackrel{\wedge }{r}}_{(t-{\omega }_1-2)}\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}+\dots +\frac{1}{{\omega }_1}+{\stackrel{\wedge }{r}}_{(t)}+\dots +\frac{1}{{\omega }_1}{\stackrel{\wedge }{r}}_{(t+{\omega }_1-2)}\mathrm{\hspace{1em}\hspace{1em}\hspace{1em}}+\frac{1}{{\omega }_1}{\stackrel{\wedge }{r}}_{(t+{\omega }_1-1)}+\frac{1}{{\omega }_1}{\stackrel{\wedge }{r}}_{(t+{\omega }_1)}$$ (2)

Finally, a simple difference calculation, using window size ω₂, was substituted in place of the derivative of the absolute value of the SEMG data during calculation (equation (3)).

$${\tau }_2=\mathrm{slope}({\mathrm{MA}}_{{\stackrel{\wedge }{r}}_t})={\mathrm{MA}}_{{\stackrel{\wedge }{r}}_t}-{\mathrm{MA}}_{{\stackrel{\wedge }{r}}_{(t-{\omega }_2)}}$$ (3)

A Central Moving Average and slope calculation occurred for the pressure data as shown in equation (4).

$${\tau }_3=\mathrm{slope}({\mathrm{MA}}_{{\stackrel{\wedge }{p}}_t})={\mathrm{MA}}_{p_t}-{\mathrm{MA}}_{p_{(t-{\omega }_2)}}$$ (4)

Seventy percent of the sampled moving difference window data was used as training input, testing and validation evenly split between the remaining 30%. All data were subdivided into these groups, training, testing, and validation randomly.

Data were classified for an action when the SEMG moving average was above the signal noise, the slope was positive over a moving window, and pressure data showed a positive value over a moving window. The SEMG noise-threshold was found by multiplying the CMA value by 1.5 (for SEMG). Hidden layer sizes of 1–10, 50, 100, and 1000 were tested. A 10 neuron hidden layer size was chosen and used for training. Training was repeated for approximately 10 times until no significant improvement in the network weights or outputs was seen. The scaled conjugate gradient method was used to update weights and bias values.⁴⁰

Data were classified using one of eight multi-layer perceptron, feed-forward neural network classifiers. Four of these classifiers included training data solely consisting of EMG data. Separately, four neural networks were trained considering each SEMG–FMG pair, for a total of eight inputs and two outputs with the result of determining if an action had occurred. Separately, two networks were trained to classify wrist deviation, one using SEMG data only and one using both SEMG and FMG data.

Training data for the first eight networks consisted of data while the subject was seated and the socket and sensors were in the “ideal” position, while the subject was standing and the socket was in the “ideal” position, and while the subject was seated and the socket had been shifted. Descriptions of these data sets can be found in the Methods: Experimental protocol section. Training of the two additional networks, as part of the experiment described in Section 2.1.4, occurred while the subject was seated and the socket was placed in the “ideal” position. Data from the experiment described in Section 2.1.3, while the subject was fatigued, was not included in any training set. A table describing which data are included in each of the eight data sets used to train the classifiers can be seen in Table 1. Column labels can be described as: EMG, FMG, “ideal” socket position, various arm positions while standing, and “shifted” socket positions. For clarity during discussion below, the classifiers will be referred to by a short-hand, concatenating the names of the data sets used to train the networks. For example, Classifier 1 will be referred to as “EMG_I”, i.e. “EMG data while the socket is in the “ideal” position” (Classifier 2-EMG&FMG_I, Classifier 3-EMG_ISt, Classifier 8-EMG&FMG_IStSh, etc.).

Table 1.

Experimental data included in each training set for each neural network classifier.

	Sensors		Socket/Arm position
	EMG	FMG	Ideal	Standing	Shifted
Classifier 1	X		X
Classifier 2	X	X	X
Classifier 3	X		X	X
Classifier 4	X	X	X	X
Classifier 5	X		X		X
Classifier 6	X	X	X		X
Classifier 7	X		X	X	X
Classifier 8	X	X	X	X	X

Results

Eight different classifiers were trained. The first two classifiers were trained using only data collected while the socket was in the “ideal” position and the subject was seated. These classifiers are used as a base or control to compare against. The first of the two control classifiers included only data from the EMG sensors. The second control classifier included EMG and FMG data. A table describing the data included in each training data set can be seen in Table 1. Data sets 1 and 2, as labeled in the above mentioned table, are the control data sets. All eight networks were tested against data while the socket was in the “ideal” position while the subject was seated, against shifted socket data, against data gathered while the subject was standing, and against data gathered during times of subject fatigue. Comparisons were also made between classifiers not including pressure data and those that included intrasocket pressure data. Final results of classification of data during times of fatigue are reported. No data from trials during times of fatigue were included in training data. Separately, two networks were trained to classify radial and ulnar deviation of the hand while the simulated socket was in the “ideal” position and the subject was seated. Figures 5(a), 7(h) and 8(a) and (b) show the confusion matrices in which the main diagonal percentages indicate the percentages of “correct” classification for flexion and extension, radial or ulnar deviation, or “no action” while off-diagonal values show the percentages of mis-classifications. Data are presented as averages of flexion and extension classification data over the entire experiment, for each classifier. The entire matrix will sum to approximately 100%, due to rounding of significant digits.

Figure 4.

Network topology of feedforward MLP used in this work. Inputs τ₁, τ₂, and τ₃ are described in Methods: Classification Algorithm.

Figure 5.

Confusion matrices comparing classifiers to standing data. Matrices a, c, e, g, used SEMG data. Matrices b, d, f, h used SEMG and FMG data. (a) Classifier 1-EMG_I; (b) Classifier 2-EMG&FMG_I; (c) Classifier 3-EMG_ISt; (d) Classifier 4-EMG&FMG_ISt; (e) Classifier 5-EMG_ISh; (f) Classifier 6EMG&FMG_ISh; (g) Classifier 7-EMG_IStSh; (h) Classifier 8-EMG&FMG_IStSh.

Figure 7.

Confusion matrices comparing classifiers to fatigue data. Matrices a, c, e, g, used SEMG data. Matrices b, d, f, h used SEMG and FMG data. (a) Classifier 1-EMG_I; (b) Classifier 2-EMG&FMG_I; (c) Classifier 3-EMG_ISt; (d) Classifier 4-EMG&FMG_ISt; (e) Classifier 5-EMG_ISh; (f) Classifier 6EMG&FMG_ISh; (g) Classifier 7-EMG_IStSh; (h) Classifier 8-EMG&FMG_IStSh.

Figure 8.

Confusion Matrix reporting classification results for radial and ulnar deviation of the hand: (a) 4 SEMG inputs; (b) SEMG and FMG inputs.

Performance against arm positions during activities of daily living

This section reports results of the eight neural networks when classifying data collected during the experiment described in Sec. 2.1.2. This experiment simulated arm positions during activities of daily living. The subject performed wrist flexion and extension motions while their dominant arm was in the four positions as seen in Figure 2(b) to (d). Figure 5(a) to (h) reports the average performance data for each classifier, for each experiment.

Performance against socket shift

Figure 6(a) to (h) reports the results of the eight neural networks when classifying data collected during the experiment described in Sec. 2.1.1. This experiment simulated socket shift as experienced when a prosthetic socket moves due to socket-pull-out or poor socket fit. The subject performed wrist flexion and extension motions while their dominant arm was resting on an arm rest. The simulated prosthetic socket was then moved, as described above, 1 cm and 2 cm away from the “ideal” socket position. Figure 6(a) to (h) reports the average performance data for each classifier, for each experiment.

Figure 6.

Confusion matrices comparing classifiers to socket shift data. Matrices a, c, e, g, used SEMG data. Matrices b, d, f, h used SEMG and FMG data. (a) Classifier 1-EMG_I; (b) Classifier 2-EMG&FMG_I; (c) Classifier 3-EMG_ISt; (d) Classifier 4-EMG&FMG_ISt; (e) Classifier 5-EMG_ISh; (f) Classifier 6EMG&FMG_ISh; (g) Classifier 7-EMG_IStSh; (h) Classifier 8-EMG&FMG_IStSh.

Performance against fatigue

Figure 7(a) to (h) report results of the eight neural networks when classifying data collected during the experiment described in Sec. 2.1.3. This experiment simulated user fatigue as experienced throughout a user’s typical day of powered prosthetic use. The subject squeezed a felt ball 10 times and then squeezed a force sensor. These motions were repeated until the force output was reduced to 80% of the initial force output. The user then performed wrist flexion and extension motions while their dominant arm was resting on a chair’s arm rest. Figure 7(a) to (h) reports the average performance data for each classifier, for each experiment.

Performance of radial and ulnar deviation classification

Figure 8(a) and (b) reports results of the trained neural network when classifying data collected during the experiment described in Sec. 2.1.4. This experiment simulated the radial and ulnar deviation of the user’s hand during activities of daily living, such as during a “hammer” or “dart throwing” action. The subject deviated their dominant wrist 10 times in the radial direction and then 10 times in the ulnar direction while their dominant arm was resting on a chair’s arm rest. Figure 8(a) reports the average performance data for the classifier for this experiment using only the four SEMG sensors as input. Figure 8(b) reports the performance data for the classifier for this experiment using the four SEMG sensors and the four FMG sensors as input. These classifiers were trained and tested against data from this experiment only; 70% of the time samples were used for training, 15% for validation, and 15% for testing.

Discussion

Flexion and extension motions were chosen for this study as a means of testing data typically available to “the greatest number of lower arm prosthetic users”. That is to say that in commercial settings with typical users, these motions have been used by protheticists for gathering signals for some time. While it is true that clinical results using the most advanced techniques available in the literature boast classification results much higher then those presented here, the classifiers trained in this study were chosen for the simplicity and ease of implementation. The results presented here are to be viewed as a baseline for future improvements. It should also be noted that although force-profiles and classifier training will be similar across a cohort of subjects, due to customization of the socket, only a single subject was considered at this time.

It can be seen from Figures 5(a) and 7(h) that in general, those classifiers trained using larger training sets, Classifiers 7 & 8 performed better than the classifiers including shorter training sets, Classifiers 1-EMG_I & 2-EMG&FMG_I. It is thought that inclusion of more data, more examples of flexion and extension motions, provides a more robust classifier. This was also confirmed by Fougner et al. during their experiments which included data collected with the subject’s arm in various positions and not just resting on a table top or a chair’s arm.

During experiments described in Sec. 2.1.2 & 2.1.1, it can be seen that those classifiers including intrasocket pressure data and SEMG outperformed classifiers with data collected from surface EMG only. This is especially true when considering Classifiers 1-EMG_I & 2-EMG&FMG_I during socket shifts. These classifiers were not trained with data collected when the socket was shifted but were tested against “shifted” data (Section 2.1.1). Interestingly, Classifiers 3-EMG_ISt and 4-EMG&FMG_ISt, trained including data while the subject’s arm was in different positions, outperformed Classifiers 2-EMG&FMG_I. Comparing Classifiers 2-EMG&FMG_I and 3-EMG_ISt, one can see that a classifier using four SEMG and four FMG sensors is nearly as accurate as a classifier including additional training data. This could have implications during initial training, implementation, and deployment for a prosthetic user. This conclusion is not necessarily true for times when a user is standing and holds their arm in different positions, however. It should also be noted that while Fougner et al. mentioned that the classifiers trained using SEMG and accelerometer data most likely took socket shift do to arm position variations in to account, magnitudes of the displacement of the sensors were not measured unlike in this work where it was a controlled variable. That is to say that position data was only recorded while the socket was in the ideal location, as described in Methods: Experimental protocol: Simulating arm positions during activities of daily living, and is meant as a means to effectively compare against the shifted socket in Methods: Experimental protocol: Simulating socket shift, where the sensor housings were displaced a controlled distance away from the ideal position. By performing a linear shift while the subject’s arm was resting on a chair’s arm-rest, EMG effects due to varied arm positions can be eliminated.

From data found in Figure 5(a) to (h), one can see that Classifiers 3-EMG_ISt - 8-EMG&FMG_IStSh performed approximately equally. But they did outperform Classifier 2-EMG&FMG_I which in turn outperformed Classifier 1-EMG_I. That is to say that classifiers including SEMG and FMG data as part of their training sets performed approximately equally to classifiers including only SEMG data when additional training data collected when the subject’s arm is in different positions, is included. But a classifier using the eight sensor setup outperforms the SEMG-only setup when additional training data is not included. This same conclusion can be made when comparing and classifying samples during times of user fatigue as seen in Figure 7(a) to (g).

No training sets included data collected during the fatigue experiment, Methods: Experimental protocol: SEMG and FMG while fatigued. This protocol was used due to hardware limitations and in an attempt to measure classification robustness as the user perspired. It should also be noted that due to the nature of the method of data acquisition and signal processing performed described above in Section 3.2 and 3.3, EMG signal frequency was not recorded. Future hardware will take this in to account, measuring the median EMG signal frequency, and using this as a means of determining user fatigue. Despite these limitations, this protocol allowed us to gather data and see that Classifiers 3-EMG_ISt - 8-EMG&FMG_IStSh were able to perform approximately equally and classify user intent. It should be noted that while the subject reported their arm “feeling tired”, they did not report “feeling sweaty”. Future experiments should attempt to force the subject to exert themselves enough to cause sweat on the skin surface in order to further test the SEMG-FMG system, since sweat has been reported to reduce the amplitude of the EMG signal. A degradation of the measured amplitude of the SEMG signal of 2–3%, depending on the type of sensor used, for every 0.02 mm of sweat between the surface of the skin and the sensor was determined in Abdoli-Eramaki et al.¹² Perspiration, a conductive fluid, can also cause intermittent short-circuits of an EMG sensor. While this can be partially compensated for as proposed by Ray and Guha,¹¹ a high frequency oscillator is necessary for signal injection, and is not used in clinical settings and the author is unaware of any production systems making use of this method. The system presented in this paper avoids EMG signal degradation issues due to sweat by making use of a multi-modal sensory input.

Prior work by Young et al. discusses classification errors based on socket shift direction along the muscle body.⁹ The authors found that socket shift and sensor displacement affected classification more readily when the socket shifted perpendicularity to the muscle fiber, i.e. rotated around the arm as compared to moved more distally or proximally. While we gathered data shifted in a similar method to Young et al., we did not control for direction of shift as part of the classifier training. Only linear distance away from an ideal location was considered. Future work may show that improvements in a reduction of training motions can be made by only rotating the socket without also moving the socket distally and proximally along the forearm.

Figure 8(a) and (b) reports findings from classification of radial and ulnar deviation motions of the subject’s wrist and hand. These motions are found during several activities of daily living, including “hammering”, “dart throwing” and many others. A significant improvement in classification can be seen between the two methods presented here, using only surface EMG and using a combination of surface EMG and force myography. It is believed that a more advanced classification algorithm could provide even further improvements. To these authors’ knowledge, classification of these actions has not been reported before in the literature. Previous work has focused on improving dexterous finger motions and classification. It should be noted that also to these authors’ knowledge, no powered prosthetic devices currently available to the public are able to provide this capability to a robotic hand. Currently, only wrist rotation actions are possible with available devices. The authors feel that the addition of radial and ulnar deviation to a powered device would improve usability and decrease long-term, compounding injuries of the remainder limb’s elbow and shoulder joints.

While fine gesture recognition for highly dexterous motions is beyond the scope of this study, it has been considered in works by Scheme and Fougner, for example. And while there has been interest in classifying data from a large array of EMG sensors, the advantages of studying smaller arrays for pattern recognition are apparent. Implementation will require real-time control of a device, making use of a micro-controller, more readily possible with fewer input signals. Sensor size must also be taken in to account when constructing a practical socket as well as actual, available muscle sites on a prosthetic user.

Advanced classification algorithms were not considered as part of this study as the goal was to test an SEMG-FMG system against an SEMG only system, using a known classification algorithm as a bench-mark. Additionally, large training sets were compared to more simple training sets. Time considerations during training of the feed-forward network with these large training sets were noted.

Conclusion

This proof of concept system has provided insight and considerations in to training data and sensor configurations for future powered prosthetic and human–robot systems. We have shown that given basic training data created by a subject seated in a chair, a system of collocated surface EMG and FMG, and a basic neural network classifier performs better than a system trained using only SEMG data by at least 3% against data collected during arm movements found in daily life. An even more significant improvement was seen against data collected during times of subject fatigue and socket shift, 8% and 35%, respectively. Additional and more complex training sets consisting of data gathered during shift of the prosthetic socket performs as well as, or better than, systems trained using data gathered during arm motions of daily living while standing. Should a user only be able to perform a single additional training data set, socket shift movements should be considered over arm movements with considerations for socket rotation. Insight into classifying data gathered during times of user fatigue was gained. And although systems making use of additional training data outperformed systems including only basic training data during fatigued tasks, no information could be gained for tasks performed while a user was perspiring at this time. Classification of radial and ulnar deviation of the wrist was shown to be possible, with a significant classification improvement of the SEMG-FMG system over the traditional SEMG only system. Future work will include implementation of DC offset compensation.¹⁴ This should further improve the captured signals. Training using the recurrent neural network method, a more computationally intensive training algorithm, is considered impractical at this time but will be considered during future work. Other more complex training algorithms will be considered and tasks designed to force a subject to exert themselves and cause perspiration will be performed.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Science Foundation NRI (grant no. IIS-1208623).

Guarantor

JS is the guarantor of this paper.

Contributorship

JS, RP, and DP contributed to the design of the study; JS was involved in data collection; JS and RP performed the data analysis; JS wrote the first draft; all authors contributed to and approved the final version.