Virtual reality environment for simulating tasks with a myoelectric prosthesis: an assessment and training tool

Abstract

Intuitively and efficiently controlling multiple degrees of freedom is a major hurdle in the field of upper limb prosthetics. A virtual reality myoelectric transhumeral prosthesis simulator has been developed for cost-effectively testing novel control algorithms and devices. The system acquires EMG commands and residual limb kinematics, simulates the prosthesis dynamics, and displays the combined residual limb and prosthesis movements in a virtual reality environment that includes force-based interactions with virtual objects. A virtual Box and Block Test is demonstrated. Three normally-limbed subjects performed the simulated test using a sequential and a synchronous control method. With the sequential method, subjects moved on average 6.7±1.9 blocks in 120 seconds, similar to the number of blocks transhumeral amputees are able to move with their physical prostheses during clinical evaluation. With the synchronous method, subjects moved 6.7±2.2 blocks. The virtual reality prosthesis simulator is thus a promising tool for developing and evaluating control methods, prototyping novel prostheses, and training amputees.

Introduction

Within recent years, dramatic improvements have been made in the mechanical design of upper limb myoelectric prostheses.¹ For instance, DEKA Integrated Solutions Corporation has developed the 10 degree-of-freedom (dof) “Luke” arm that has entered clinical trials.² In contrast, a state-of-the-art commercially available myoelectric transhumeral prosthesis has 3 dof (elbow flexion, wrist rotation, and a terminal device) that is generally operated sequentially by using a switch command between the operation of individual dofs. A major unsolved problem in upper limb prosthetics is providing reliable independent command sources for intuitively and efficiently controlling multiple degrees of freedom simultaneously. Virtual reality has been suggested as a method to quickly develop and evaluate control strategies, prototype devices, and train subjects.^2–6 Another recent study demonstrated that training with a virtual prosthesis was equivalent to training with a physical prosthesis.⁷

Previous myoelectric prosthesis simulators have included costly or complicated components making them impractical for widespread use in clinical settings.^{4, 5} In addition, while simulators have accurately modeled the dynamics of the prostheses themselves,^{4, 5} modeling of interactions with the virtual environment has been limited. Modeling of the behavior of the prosthesis under variable loads and describing the characteristics of objects to be grasped are necessary, but so far unachieved goals, to demonstrate functionality of a prosthesis in a virtual environment.⁵ Advances in computer hardware and development of real-time physics simulation software—driven by widespread use in commercial video games—have made these goals realizable.

This article describes the software and hardware components of a simulator for evaluating and training novel command and control strategies for transhumeral myoelectric prostheses. The system is composed of portable easy-to-setup components that can be taken home by a user, but which still realistically simulate functional tasks via real-time physics simulation. A virtual version of a common clinical assessment test was implemented as a demonstration, in which two prosthesis control methods were compared.

Methods

SYSTEM OVERVIEW

The subject (intact-limbed or transhumeral amputee) views an animation of his or her humeral movement and the simulated prosthesis movement in a virtual environment. The system includes several components (illustrated in Figure 1, and described in detail below) which are interfaced to a single laptop or desktop PC, running MATLAB/Simulink (The Mathworks, Inc. Natick, MA) and a custom virtual environment application.

Figure 1.

Schematic of the Transhumeral Prosthesis Simulator. The subject performs simulated tasks in a virtual environment that incorporates movements of the actual upper limb, EMG command acquisition, and dynamic modeling of the commanded prosthesis and its interactions with virtual objects.

1. KINEMATIC TRACKING

Kinematic tracking of the upper limb is achieved using a 3DM-GX1 Orientation Sensor (MicroStrain, Inc., Williston, VT). This sensor combines tri-axial accelerometer, tri-axial magnetometer and gyroscopes to provide an accurate 3D orientation with respect to an Earth-referenced coordinate system. Gyro-stabilized ‘ZYX’ Euler angles are polled from the sensor in Simulink at 20 Hz. Before using the simulator, a reference coordinate system (A_ref), about which rotations will be calculated, must be determined. The sensor is held near the upper arm with a predefined orientation. Polled Euler angles are converted to a rotation matrix and stored. Next the sensor is attached to the upper arm with a strap and the subject is asked to hold the arm in the anatomic position (humerus parallel to trunk with palm facing anteriorly). A rotation matrix (R₀) for that posture is similarly calculated and stored. All subsequent rotations are calculated relative to this nominal orientation using equation 1, where R_out is the rotation matrix used to define the orientation of the upper arm and R_in is the rotation matrix computed from the raw sensor values.

$${\mathbf{R}}_{\mathit{out}}={{\mathbf{A}}_{\mathit{ref}}}^{\text{T}}\left({\mathbf{R}}_{in}{{\mathbf{R}}_0}^{\text{T}}\right){\mathbf{A}}_{\mathit{ref}}$$ (1)

2. EMG ACQUISITION

Electromyogram (EMG) signals are acquired and amplified using disposable surface electrodes and a BioRadio 150 wireless 12-channel physiological signal monitor (Cleveland Medical Devices, Inc., Cleveland, OH). Up to 8 channels can be configured to record EMG at 960 Hz. EMG data is digitally high-pass filtered—to remove motion artifact—and processed in packets in Simulink. The setup allows for unlimited customizability of the EMG processing/command algorithms. The device also has an auxiliary input that can be used to interface with an external push button or switch. More details on the specific EMG setup used in the experiment is included below (Experimental Design).

3. PHYSICS SIMULATION

Physics simulation is implemented using Newton Game Dynamics (NGD, newtondynamics.com), a deterministic force-based solver. Massera and colleagues used NGD to simulate an anthropomorphic neurorobotic arm to study the ability of evolutionary algorithms to achieve functional grasping patterns.⁸

The prosthesis in the virtual environment was modeled after the Utah Arm 3 (Motion Control, Inc., Salt Lake City, UT). The prosthesis has three independently controlled dof: elbow flexion/extension, wrist pronation/supination, and grasp aperture. The fingers and thumb are subject to a kinematic constraint such that, during grasp the thumb moves twice the angle of the fingers. Dynamic properties of the simulated prosthesis were tuned to mimic the behavior of the physical prosthesis. Figure 2 illustrates the joint axes and the collision/inertial hulls for the forearm, hand, finger, and thumb segments. The segment coordinate systems are shown at each segment’s center of mass.

Figure 2.

Posterior view of the right arm illustrating the 7 joint axes (cylinders), collision hulls, and segment coordinate systems used in the physics simulation. The 3 joint axes at the shoulder are controlled kinematically by the actual upper limb motions, and the 2 joints in the hand are constrained by a single grasp command, resulting in 3 commanded dofs (elbow flexion-extension, forearm pronation-supination, and hand aperture).

The simulated residual limb segment is kinematically constrained to match the orientation computed from the orientation sensor. The requirement for accurate placement of the upper limb based on the subject’s limb orientation and simulation of physical interactions in the environment can result in a paradox. Since the virtual environment may limit the virtual arm due to an obstacle, but cannot limit the user’s actual movement, the user could place his/her arm in such an orientation that the virtual joint constraints cannot be maintained. The kinematically controlled shoulder joint was therefore made less stiff than the prosthesis joints to allow the virtual shoulder joint to “dislocate” slightly (i.e., not exactly match the user’s true posture) to help insure that the prosthesis constraints can be maintained. Users were also instructed to avoid upper limb movements that would “jam” the prosthesis into walls and corners in the virtual environment.

Objects in the environment are treated as rigid bodies and can take any shape. NGD supports various collision hulls including boxes, ellipsoids, cones, cylinders, or any convex shape. Compound hulls allow multiple collision hulls to be combined to form complex shapes that include concavity (e.g. a mug with a handle), (See Fig 3). The use of object “materials” allows for varying frictional coefficients between surfaces. For example, the friction between the hand and the blocks was set higher than the friction between the blocks and the box, or between the hand and the box, to more closely reflect the interactions between a myoelectric hand and physical objects.

Figure 3.

Screenshots from the Prosthesis Simulator. The top panel shows the virtual Box and Block Test. The bottom panel illustrates the variety of virtual objects that can be simulated and manipulated.

4. VISUALIZATION

The virtual environment graphics were created using Gamestudio A7 game development system (Conitec Datasystems, Inc., La Mesa, CA). The virtual environment includes realistic features expected in modern video games: soft skin deformation, dynamic shadows, and high quality 3D graphics (See Figure 3). The virtual reality simulator is displayed using the NVIDIA 3D Vision System (NVIDIA Corporation, Santa Clara, CA) to provide depth perception and enhanced immersion. The 3D Vision system consists of a 120 Hz computer monitor—which alternately displays left and right eye images—and active shutter glasses that are synchronized to the monitor by an infrared signal to produce a compelling steroscopic view of the environment and the simulated arm.

The virtual camera is placed at the location of the virtual person’s eyes to provide a first person perspective. A common limitation of simulators is that a single display monitor does not provide sufficient viewing angle to mimic peripheral vision. Flight simulators for example commonly use multiple monitors placed around the subject. Other simulators ⁴ and video games, particularly with head-mounted-displays, use head orientation tracking to allow the subject to move the virtual camera to point towards a region of interest. Instead, to simplify our setup, an automatic camera tracking algorithm is used to keep the camera pointed towards the region of interest—assumed to be the hand. The camera direction vector is calculated as a weighted average of the tracking vector and the nominal sight vector, where the tracking vector is a unit vector in the direction from the eye position to the center of the hand, and the nominal sight vector is a unit vector pointing anteriorly and inferiorly. In this manner, the virtual camera always keeps the hand within the view but also maintains a natural forward facing view.

The virtual reality environment and physics simulation are compiled (“published”) together as a standalone executable application. Virtual prosthesis commands and residual limb kinematics are sent to the application from MATLAB/Simulink using shared memory mapping.

EXPERIMENTAL DESIGN

The prosthesis simulator is intended to be a customizable platform for evaluating command and control algorithms and for training and evaluating the performance of amputees using various prostheses and controller controller algorithms. For this study, as a demonstration, a modified Box and Block Test was simulated, and two control methods were evaluated. In the Box and Block Test of Manual Dexterity,⁹ subjects are asked to move as many blocks from one side of a box to another over a divider within one minute. Dimensions in the virtual world were based on the standardized Box and Block dimensions,⁹ except for the depth of the box (2.2 cm instead of 7.5 cm). Due to the difficulty of the task for transhumeral amputees, the time period was extended to two minutes and the task was performed standing rather than seated.¹⁰

Table 1 summarizes the command sources used for each prosthesis action using a sequential and a synchronous command method. In the sequential method, the EMG signals from an antagonist pair of muscles, biceps and triceps, were used to control the prosthesis. A push button located on the clavicle was used to switch between each of the three actions. In the synchronous control method, each prosthesis function was commanded by a different muscle, allowing multiple functions to be controlled simultaneously. The upper trapezius and latissimus dorsi were able to be independently activated by subjects while elevating and retracting the shoulder, showing potential for controlling the hand and wrist. Because of the limited number of additional available EMG sources, the hand was configured to automatically close and the wrist was limited to rotate in one direction.

Table 1.

Command sources for prosthesis actions using Sequential and Synchronous Methods. A chin operated push button allowed switching between modes when using the Sequential method

Method	Elbow		Wrist		Hand
	Flexion	Extension	Supination	Pronation	Closing	Opening
Sequential	biceps	triceps	biceps	triceps	biceps	triceps
Synchronous	biceps	triceps	upper trapezius	none	automatic	latissimus dorsi

Three normally-limbed subjects gave informed consent to participate in the study. Two disposable snap-type surface electrodes were placed approximately 2.5cm apart on each muscle (from Table 1) parallel to the fiber direction—estimated using anatomical images as a guide—in order to record a differential signal. An additional reference electrode was placed on the elbow. All electrodes were connected to the BioRadio 150 with approximately 1m long leads. EMG was digitally high-pass filtered (20 Hz 3^rd order Butterworth) to eliminate motion artifact and notch filtered (59–61 Hz 4^th order Butterworth) to reduce 60Hz noise. For both command methods, EMG data was processed by taking the mean rectified value of the EMG over 50 ms packets and applying a gain factor and minimal activity threshold (“deadzone”). Appropriate EMG gains and thresholds were determined prior to testing such that a maximum command could be achieved with slightly less than a maximum voluntary contraction, and that the upper arm could be moved around without eliciting an unwanted command.

Prior to testing, each subject practiced for at least 20 minutes with both command methods to become familiar with the simulator and the command methods. The command method tested first was randomly selected for each subject. The virtual Box and Block Test was completed five times using the first command method with 5-second pauses between each 2-minute test. The subject then rested, refamiliarized with the second command method, and completed five more tests in the same manner. This set of tests was repeated once in the same order, such that each subject completed a total of ten virtual Box and Block Tests for each command method.

Results

Three normally limbed subjects completed a virtual modified Box and Block¹⁰ test 10 times for the sequential and synchronous methods. Results are shown in Figure 4 alongside results reported by Miller et al for transhumeral amputees performing the same test with a physical prosthesis, using conventional prosthesis command methods. The “conventional” command methods used by each amputee were optimized by prosthetists and occupational therapists, and may not have been identical for each subject.¹⁰ Although not identical to either command method used in our simulator, the conventional command methods used were sequential in nature. On average in the simulator, subjects moved 6.7±1.9 blocks using the sequential method and 6.7±2.2 blocks using the synchronous method. The amputee subjects moved, on average, 6.8±3.1 blocks using a physical prosthesis.

Figure 4.

The right panel shows the mean (n=10) number of blocks moved in the virtual Box and Block Test using two different control methods (error bars indicate 1 SD) for three normally-limbed subject. The left panel shows the results (n not reported) for three transhumeral amputees performing the physical Box and Block Test using conventional prosthesis control methods (as reported by Miller et al, 2008)

A repeated measures 2-way ANOVA was performed on the Box and Block data, with control method and test set as factors. The test set—first or second set of 10 tests— was considered a factor to determine if there were significant improvements (due to learning) or worsening (due to fatigue, attention-loss, etc.) in performance. Across both control methods, the average number of blocks moved was 6.5±2.2 in the first set and 6.9±1.9 in the second set No statistically significant differences were found between the control methods (p=0.98) or the test sets (p=0.64).

While no statistical claims can be made on the closeness of the sequential (conventional) simulated results to the physical prosthesis results reported by Miller et al—because the number of times each transhumeral subject was tested was not reported—it is evident that the results are very similar. However, it seems that the physical prosthesis results were substantially more variable than the simulated results (as determined by larger standard deviations within and between subjects).

Discussion

IMPLICATIONS

A portable, cost-effective, and simple to use virtual reality myoelectric prosthesis simulator has been developed that accurately simulates both the dynamics of a transhumeral prosthesis and its interactions with the environment. The simulator allows for assessment in a virtual environment, such as with the Box and Block Test demonstrated in this study. However, virtual objects are not limited to blocks, and can take any shape and size. Objects can also vary in weight, elasticity, and surface friction. The simulator allows clinicians and engineers to easily test new command and control algorithms and prototype novel devices in a functionally relevant manner. Many studies have reported on the classification accuracy of various pattern recognition-based control algorithms, but the functional relevance of their results is unclear. The simulator allows for functionally relevant training of an amputee before fitting and receiving a prosthesis. Furthermore, the portability of the setup would allow the amputee to train outside of the clinic. Another major advantage of the virtual simulator is that normally-limbed subjects can be used to test algorithms.

In the simulator, kinematic recording of the residual limb is required as most manual tasks require positioning the whole arm, not just the joints of the prosthesis. Positioning of the residual limb also has other implications: the biceps and triceps—commonly used as command sources in transhumeral prostheses—are both biarticular muscles that normally cross both the elbow and shoulder joint and therefore may become active when the shoulder is moved. This muscle activity could elicit unintended commands. Therefore, it is necessary to test control algorithms with various shoulder angles. Micera and colleagues highlighted the importance of posture on decoding in a recent review.¹¹ By implementing kinematic recording of the residual limb, the prosthesis simulator allows evaluation of control algorithms in all possible postures.

LIMITATIONS AND FUTURE WORK

A major limitation of the virtual reality simulator is that the user does not feel the movements of the virtual prosthesis. The weight and inertial effects of the prosthesis alone could affect the muscle activity recorded from the residual limb. In addition, when collisions occur in the virtual environment, the user’s real arm is unaffected which can sometimes lead to a paradox (described in the Methods – Physics Simulation), where the user can force the prosthesis into impossible configurations. Haptic feedback systems, such as the HapticMaster,¹² could be used to provide a virtual haptic interface in combination with the virtual visual environment. The device could exert an opposing force on the user’s real residual limb when the user’s movements or commands cause a collision in the environment, eliminating this paradox. The loading from the weight of the prosthesis and any grasped objects could also be simulated. The requirement to develop a simple, portable, and cost-effective simulator limited the ability to use haptic devices in this study. Some prostheses utilize tactile vibrators to indicate the pressure being applied to a grasped object. Such devices could easily be incorporated into the simulator as well.

The quality of kinematic tracking depends largely on the 3-axis magnetometer output. A room with steel file cabinets, desks, or chairs may have substantial fluctuations in the measured direction of North even over the small distances that the orientation sensor translates when mounted on the residual limb. This results in inaccuracies in the recorded rotation about the vertical axis. It was therefore important to insure that the volume in which the residual limb moved had a relatively constant magnetic field by monitoring the output of the sensor as it was translated through this volume with a fixed orientation. It was also necessary to define the reference coordinate system within this volume. In this study, the subject stood at least 1.5m away from the nearest ferrous metal object.

Due to the real-time requirement of the simulation, physical interactions between objects must be limited to some extent. On a Dell Dimension dual core 2.4 GHz PC with NVIDIA 9800 GTX graphics card, the simulation ran near the maximum 100 frames per second except when multiple objects were being constrained simultaneously. For example, if multiple blocks were being pushed into the divider simultaneously, the framerate dropped dramatically causing instability in the simulation. Therefore, only one block at a time was placed, in a random position and orientation. When the user successfully moved the block to the other side, a new block appeared. Since the box is normally filled with 150 blocks,⁹ the top layer of blocks is near the top edge of the box. The virtual box had a decreased depth to mimic this and reduce the likelihood of impossible-to-resolve collisions.

The currently simulated prosthesis has a conventional hand with a single dof, grasp aperture. The advent of multi-articulated hands with multiple grasp patterns such as the commercially available iLimb (Touch Bionics, Edinburgh, Scotland), bebionic hand (RSLSteeper, Rochester, England) and soon to be available Michelangelo hand (Otto Bock, Duderstadt, Germany), has surely revolutionized prostheses. Future work will incorporate these sophisticated hands into the virtual environment to aid in the development of control methods. In addition, for a transradial prosthesis simulation, the elbow flexion and forearm pronation angles will need to be measured to kinematically constrain the distal residual forearm segment. This can be achieved with an additional orientation sensor.

Both the prosthesis and the objects with which it interacts can be modified in the simulator. Future work may involve developing more virtual functional assessment tasks such as the clothespin relocation task,⁴ Nine-hole peg test of finger dexterity,¹³ Grooved Pegboard Test,¹⁴ or possibly the Southhampton Hand Assessment Procedure (SHAP)¹⁵ which incorporates bimanual tasks.

Conclusion

The feasibility of using a virtual reality prosthesis simulator for training and assessment has been demonstrated with a virtual Box and Block Test. The virtual test was compared with data from amputees performing the same test with a physical prosthesis. Our simulator approach should allow for functionally relevant testing of the command and control algorithms presented in the literature. The transhumeral prosthesis simulator presented here could easily be modified to incorporate transradial and shoulder disarticulation prostheses, or any novel multi-dof devices. Simulated interactions with virtual objects of various shapes, sizes, and weights allow virtual representation of almost any task. Furthermore, the simulator system is simple to setup, portable, and relatively inexpensive which may prove important in the acceptability for widespread clinical use in amputee training.